Research Skill Is Training in Problem Selection and Feedback Loops
When AI can cheaply generate mediocre answers, core research capability lies in having the taste to ask essential questions and building feedback loops driven by multi-perspective debate, rather than hoarding information.
Both *how to be good at research* and the Stanford STORM methodology treat research as a dynamic feedback system. My takeaway from reading them is that simply retrieving materials is not enough; we need to build the feedback process itself into the research.
Right now, the cost of calling a large language model to generate a well-structured, authoritative-sounding, and seemingly detailed literature review or industry report has dropped very low. Yet text generated this easily is often little more than a "mainstream average answer." It aggregates the most common narratives on the internet while smoothing over real-world friction, competing interests, and counter-evidence.
If we understand research skill merely as "collecting and reading large volumes of material," that skill will quickly lose value in front of generative models. The gap in research caliber shows up in two capabilities: the taste for choosing essential questions and building feedback loops of multi-perspective debate and falsification. You cannot measure these capabilities simply by looking at how much information someone possesses.
Choose Your Own Questions, Refuse to Passively Inherit Hot Topics
When starting a research project, the most common mistake many people make is passively absorbing topics handed to them by others, such as viral social media debates, keynote slides just announced by a major industry player, or a trendy concept everyone in their circle is chasing.
As I understand it, the biggest danger in casually picking up such topics is that you only receive someone else's pre-packaged conclusions without working through the underlying causal reasoning and assumptions. Once the foundational conditions of the original narrative drift, trend-followers often realize only after a long delay that the logic has collapsed.
The mathematician Richard Hamming once asked a famous question: "What are the most important problems in your field, and why aren't you working on them?"
Inspired by this question, I examine a topic item by item before gathering any materials:
- Why does this issue have to be solved? Is the answer addressing a real problem, or an artificially manufactured pseudo-demand?
- If I reach a firm conclusion, who will change their behavior as a result? Can it guide concrete engineering, product, or investment actions?
- Why is it worth doing at this exact moment? What new variables have emerged (technical maturity, cost inflection points, policy boundaries)?
- Do I have a distinct vantage point or authentic first-hand data?
- Most importantly: if my core hypothesis is wrong, what kind of contrary facts would I expect to observe?
A question is only worth investing energy to dig into if it has clear real-world impact and explicit boundaries for falsification.
Engineer Multi-Perspective Clashes: Conflict Has Higher Information Density Than Summaries
Many researchers are used to stitching together paragraphs from different articles into a summary, but I think this often obscures what were originally sharp contradictions. Before jumping into fact retrieval, actively designing a multi-perspective clash is a more effective path.
An industry or technical question worth investigating usually interweaves several competing stances:
- Frontline practitioners: care about concrete engineering implementation, operational friction, delivery cycles, and hidden costs, while remaining naturally cautious toward grand narratives;
- Theoretical researchers: strictly examine the internal consistency of deductive logic, the precision of conceptual definitions, and the reliability of academic evidence;
- Skeptics: specifically target the blind spots, extreme edge cases, and historical failures that mainstream optimistic narratives deliberately avoid;
- Stakeholders: ask whose profits this new technology or business model actually shifts, and who is left holding the risk;
- The historical-cycle view: zooms out to check whether the current frenzy is merely a fresh replay of an old paradigm driven by compute or capital.
Putting these stances into the same framework to collide quickly exposes four core types of conflict:
- Factual conflict: Are there obvious contradictions in the data definitions across different sources?
- Interpretive conflict: Facing the same industry phenomenon, why do different roles offer vastly different causal mechanisms?
- Time-horizon conflict: Are operations that bring high short-term profits quietly accumulating systemic risk over the long run?
- Conflict of interest: Do advocates claiming to represent the future build their core business models on high premiums at a specific link in the chain?
In my experience, sorting out these conflicts yields far more information than reading ten cookie-cutter industry overviews.
Build Your Own Evaluation System (Eval)
As large model training has evolved, people have gradually realized that stacking massive parameters during pre-training only lays a semantic foundation. Whether a model can ultimately follow complex instructions and demonstrate rigorous logic depends at its core on how Graders, Rewards, and strict evaluation (Eval) sets are designed during post-training.
The same principle applies to personal research. Without a rigorous evaluation mechanism, research degenerates into aimless material hoarding.
I believe every core thesis must be tested against an internal Eval:
- Which current anomalies can this judgment explain? And what can it not explain?
- What kind of hard data or real-world evidence would make me immediately revise or even overturn this judgment?
- Is this conclusion built on verifying first-hand financial reports, source code, and test results, or does it merely repeat the spoken predictions of an industry authority?
An opinion that cannot be refuted and lacks clear falsification conditions has no practical value in engineering or business decisions.
Writing Is External Memory for Thinking
Research relies on writing to turn thought into records that can be traced back and compared. You cannot complete this process by letting ideas spin in your head and then trying to write out the conclusion in a single breath.
In my daily notes, I prefer to organize my knowledge system as a continuously updated network of questions and evidence:
- Collected raw materials and scraped data are always isolated as pure evidence;
- Every distilled core judgment comes with the chain of evidence it stands on and an entry point for counter-evidence;
- When multiple judgments intersect in a domain, I organize them into a structured topic page that marks currently known facts, contested disagreements, and the next verification questions to follow up on.
I organize these records into a map of judgments that absorbs new evidence and corrects errors as the real world changes. Research assets built this way remain usable over time, something a static article that expires upon publication can rarely achieve.
Boundaries and Caution
When using multi-perspective clashes and logical deduction, I remind myself to keep them separate from factual evidence.
Model-assisted multi-perspective deduction can help us broaden our horizons and spot blind spots in our thinking, but it is not the same as factual evidence. It cannot replace reading first-hand official audit reports, running code assertions in a local terminal, or verifying an industrial supply chain on the ground.
When facing high-impact practical decisions, my rule is that all logical deductions must ultimately converge back onto objective hard evidence to be tested.