Evaluating the quality of evidence for gaming disorder A summary of video game addiction journal articles systematic reviews of associations between gaming disorder and depression or anxiety
Evaluating the quality of evidence for gaming disorder A summary of video game addiction journal articles systematic reviews of associations between gaming disorder and depression or anxiety
A second major concern is that reviews did not limit evidence synthesis and conclusions to studies that measured the construct of gaming disorder and at times used overly-broad definitions of depression and anxiety , which might have led to reports of associations between gaming disorder and depression or anxiety when none might exist. Based on our definition for measurement of gaming disorder , no review focused only on studies that measured gaming disorder. The percentage of studies within a review that measured gaming disorder ranged from 56.8% to 93.6%. On sensitivity analysis, where measurement of gaming disorder also included using an Internet addiction scale in a gaming disorder clinical population, the percent remained similar, ranging from 58.6% to 93.6%. Data Availability: All relevant data are within the manuscript and its Supporting Information files. Data on depression and anxiety consisted of study scale, type of analysis, direction of association , and how each review reported the outcome of the study . We classified a review as reliable only if all six of the criteria were met. Finally, we conducted a full assessment of the quality of the included reviews using A Measurement Tool to Assess systematic Reviews—version 2 ; the full results of this assessment are found in S1 Data Extraction . Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: Dr. Colder Carras has spoken to the World Health Organization about this topic at the request of the Entertainment Software Association but has received no funding, honorarium, fees, meals, lodging, donations or reimbursement. She has also spoken about this topic to other audiences: She received an honorarium and airfare to speak to the Johns Hopkins Department of Psychiatry about video games and mental health and has addressed this topic in other unpaid/unreimbursed conferences and lectures. Dr. Colder Carras has no past, current, or planned business or financial relationships with any other organization related to the video game industry. She has acted as a consultant with a nonprofit organization, Stack Up, that provides mental health and suicide prevention support through online video game play. She is currently a member of the American Association for Suicidology Technology and Innovation Committee and believes that video games and social media have the potential to be useful platforms for delivering public health interventions. She sometimes plays video games herself. The other authors declare no conflict of interest. Six studies reported additional cross-sectional depression results that were not incorporated into any review. Three of these studies reported null findings and in one of those cases, results were null in both bivariable and multivariable analyses. An additional 15 studies were mentioned by reviews as reporting associations between gaming disorder and depression, but using domain definitions in Figs 2 and 3 , these were not found . All but one of the six reviews that included these studies reported these as positive associations. Some reasons for this were: studies used a measure of Internet addiction or other exposure , studies reported a composite measure as depression, and possible mistake in citation or data extraction . Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing – original draft, Writing – review & editing Editor: Florian Naudet, University of Rennes 1, FRANCE Only one study reporting an association between gaming disorder and depression examined both cross-sectional and longitudinal associations, and the results were inconsistent . However, the results were incorporated into two of three reviews as showing a positive association. The final review used a composite definition when reporting associations, which made comparisons difficult . Gaming disorder has been described as an urgent public health problem and has garnered many systematic reviews of its associations with other health conditions. However, review methodology can contribute to bias in the conclusions, leading to research, policy, and patient care that are not truly evidence-based. This study followed a pre-registered protocol with the objective of identifying reliable and methodologically-rigorous systematic reviews that examine the associations between gaming disorder and depression or anxiety in any population. We searched PubMed and PsycInfo for published systematic reviews and the gray literature for unpublished systematic reviews as of June 24, 2020. Reviews were classified as reliable according to several quality criteria, such as whether they conducted a risk of bias assessment of studies and whether they clearly described how outcomes from each study were selected. We assessed possible selective outcome reporting among the reviews. Seven reviews that included a total of 196 studies met inclusion criteria. The overall number of participants was not calculable because not all reviews reported these data. All reviews specified eligibility criteria for studies, but not for outcomes within studies. Only one review assessed risk of bias. Evidence of selective outcome reporting was found in all reviews—only one review incorporated any of the null findings from studies it included. Thus, none were classified as reliable according to prespecified quality criteria. Systematic reviews related to gaming disorder do not meet methodological standards. As clinical and policy decisions are heavily reliant on reliable, accurate, and unbiased evidence synthesis; researchers, clinicians, and policymakers should consider the implications of selective outcome reporting. Limitations of the current summary include using counts of associations and restricting to systematic reviews published in English. Systematic reviewers should follow established guidelines for review conduct and transparent reporting to ensure evidence about technology use disorders is reliable. Ten of the 31 studies reporting associations between gaming disorder and depression reported both bivariable and multivariable analyses. In five of these 10 studies, results from both analyses were robust and positive, while five studies reported inconsistent results. Whether consistent or inconsistent in the studies, positive results were incorporated into five of the six reviews that included depression findings from the study. Is the Subject Area "Medical risk factors" applicable to this article? Yes No To advance the field of addictive behaviors and ensure that research measures and reports constructs rigorously and with clarity, existing standards for systematic review conduct and reporting should be followed. Increasing transparency of reviews and minimizing the risk of bias requires the effort of multiple agents. Authors must prospectively register protocols ; use reporting guidelines, such as those from the EQUATOR Network; and share data, analysis code, and other study materials. Journals and editors must verify authors’ adherence to reporting guidelines . Although public health decision-making should always proceed on the best available evidence , the data provided in this paper suggest that limiting technology-related diagnoses to video game play is not likely to accurately reflect the findings of years of research surrounding problematic technology use. A highly rigorous systematic review that fully specifies outcome domains is needed to clarify the potential mental health problems associated with problematic technology behaviors, including video gaming and Internet use. Roles Data curation, Investigation, Methodology, Project administration, Supervision, Validation, Writing – original draft, Writing – review & editing A third major concern is that only one review reported a systematic assessment of risk of bias using multiple domains, which has long been a best practice in conducting systematic reviews . When the risk of bias is not systematically assessed and reported, conclusions from studies included in reviews may be seen as valid and reliable when they may actually reflect biases, such as selection bias, information bias, and/or confounding . When evidence of questionable methodologic quality is used to inform public health or policy decisions, such decisions may be misguided. For more information about PLOS Subject Areas, click here . Is the Subject Area "Games" applicable to this article? Yes No Is the Subject Area "Systematic reviews" applicable to this article? Yes No In the sensitivity analysis, one additional study reported inconsistent associations in bivariable and multivariable analysis and was reported as positive in the one review that contained it. IGD = Internet gaming disorder; PG = problematic gaming; PIU = problematic Internet use. Sensitivity analysis: Clinical population of those seeking help for gaming-related problems but an Internet addiction scale was used. Including those adapted from Internet addiction scales where an example question is given. Where scales referenced appendices or other papers, these were also searched for example questions. The current overview is subject to certain limitations. First, at the level of the studies we found significant inconsistencies in measurement and analysis, which were dealt with by describing counts of associations by type. While this is a somewhat reductionist approach to summarizing results, it helps paint a picture. Relatedly, no reviews defined outcomes completely. Second, we limited our analysis to systematic reviews published in English. It is possible that our findings may have been different had we included reviews in other languages. Third, we focused on the outcomes of depression or anxiety. This narrow scope made a detailed analysis possible, but findings regarding associations between gaming disorder and other outcomes may have been different. However, due to the ubiquitous nature of selective outcome reporting, in particular, in the reviews herein, we consider this to be unlikely. Fourth, we defined the constructs of gaming disorder, depression, and anxiety very specifically; had we used broader definitions, our findings would likely be different. However, using a narrow definition was our aim. We do not attempt to draw conclusions at the study level due to the inconsistency within studies and the uncertain nature of the examined evidence. Finally, in our search of PubMed we used the PubMed publication type filters of “systematic review”, “review,” or “meta-analysis”, while we broadened our search of PsycInfo to include these terms as text-words in all fields. For this reason, it is possible that we missed some systematic reviews that were only available in PubMed and were not indexed using these terms or did not contain these terms in the title, abstract, publication type, or keywords. See section above entitled ‘Assessment of reliability of reviews’. Affiliation Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, and Department of Epidemiology, Brown University School of Public Health, Providence, Rhode Island, United States of America For more information about PLOS Subject Areas, click here . The authors are grateful to Michael M. Hughes for assistance with the graphic design and formatting of figures for publication. Although more recent reviews had higher proportions of gaming-only measures, even recent reviews included studies that used Internet addiction questions to measure gaming disorder. Distinguishing between problematic behaviors is vital in ongoing research of problematic technology use and will continue to be relevant to shaping the future of health policy and government regulation of the Internet, video games, and other forms of media and technology. Ensuring that systematic and accurate measurement of gaming disorder in studies and accurate measurement and reporting of exposures, outcomes, and conclusions in reviews are vital to inform ongoing decision making regarding diagnosis, treatment, and public health interventions. We narratively describe the characteristics of included reviews and their reliability. Because measurements of exposures and outcomes were heterogeneous, we present counts of positive or null/negative outcomes from studies and how they were reported in reviews . Because consistency is one factor that supports strength of evidence, we compared tallies of qualitative associations from the multiple outcomes reported in studies. We described associations to be ‘positive and consistent’ at the study level if the count of statistically-significant positive associations was greater than the total number of negative or null associations. We described an association as ‘null’ if there were more null findings or negative associations than positive. We conducted a sensitivity analysis to examine the impact of measuring gaming disorder with a scale for Internet addiction in a clinical population of individuals with gaming disorder. All extracted data and derived variables are available in S1 Dataset . We examined reviews that included studies of the associations between the exposure of gaming disorder and the outcomes of depression or anxiety. We restricted to reviews published in English by June 24, 2020. We excluded reviews that: Were narrative reviews, overviews of reviews, commentaries, and other non-systematic reviews of studies; Only examined Internet addiction or other technological addiction; or Did not report results for the associations between gaming disorder and depression or anxiety separately . The characteristics of the seven included reviews are reported in Table 1 . They included a total of 196 unique studies. The number of included studies per review ranged from 24 to 63, with a mean of 46. Most studies were included in only one review each. The PRISMA checklist for the current study is available in S1 PRISMA Checklist . This study was conducted using publicly-available information and therefore did not require Institutional Board approval. In a sensitivity analysis that included studies where a broad Internet addiction scale was used to measure gaming disorder in a clinical population identified as having gaming disorder, one additional study was found to have positive associations and was reported in the single review that included it as positive, while another three studies had null findings which were not reported by the three reviews that included them. Is the Subject Area "Video games" applicable to this article? Yes No Evaluating the quality of evidence for gaming disorder A summary of video game addiction journal articles systematic reviews of associations between gaming disorder and depression or anxiety
Evaluating the quality of evidence for gaming disorder A summary of video game addiction journal articles systematic reviews of associations between gaming disorder and depression or anxiety
We developed and pilot tested a data extraction form using Microsoft Excel ® , based on the form developed by Mayo-Wilson et al. . We added questions relevant to reviews of epidemiological studies . During the initial data extraction, we noticed discrepancies in how specific studies were reported in the reviews, resulting in potential selective outcome reporting at the review level. To ensure that we evaluated this potential source of bias, we expanded the scope of our preregistered protocol to include examining study-level outcomes and how they were reported in reviews. Roles Investigation, Methodology, Resources, Supervision, Writing – review & editing Is the Subject Area "Metaanalysis" applicable to this article? Yes No We found that none of the seven included reviews fulfilled all six criteria for reliability. All reviews defined eligibility criteria and most reviews conducted comprehensive database searches . No review defined outcomes using all five elements of completely-specified outcomes . No reviews specified which outcomes of a study would be used in synthesis. One review specified that it would consider only study effect sizes from multivariable analyses, classifying full associations as “…a correlation was found for both genders after multivariable analyses” or partial associations as “…correlation was identified for only one gender” . Other reviews did not specify how outcomes would be included, although some mentioned that "factors", "disorders", "comorbidity", "health-related outcomes", or "psycho-social features" "associated with" problematic gaming were "identified" , "ascertained" , or "extracted" . Overall, no review satisfied all the criteria we used to identify reliable reviews, so we could not address this research question. a = Composite reporting of outcomes in review made comparisons difficult. Of the 28 studies that reported associations between anxiety and gaming disorder, results from only 22 of these studies were incorporated into reviews . PLOS is a nonprofit 501 corporation, #C2354500, based in San Francisco, California, US This summary of systematic reviews found methodological problems in all seven systematic reviews that reported on associations between gaming disorder and depression or anxiety; no reviews could be classified as reliable based on established criteria. Although most systematic reviews studied herein defined their criteria for selecting studies and conducted a comprehensive search, each review was rated as unreliable because of one or more of the other criteria. Because of the poor pre-specification of how outcomes would be included, it is difficult to draw conclusions from these reviews regarding associations between gaming disorder and depression or anxiety that are supported by evidence. These findings suggest that the way systematic reviews of gaming disorder have been reporting results and drawing conclusions may have introduced bias into the gaming disorder literature, possibly misleading future research, policy-making, video game addiction journal articles and patient care. Delineation and measurement of a clear construct with no overlap with other related conditions, such as gambling, Internet use, and technology use, are crucial to this field. Many recent commentaries on the need for a diagnosis of gaming disorder use terms like “Internet addiction or Gaming disorder” , “Internet-related disorders including gaming disorder” , and “Internet addiction including gaming addiction” , pointing to the persistent overlap in measurement of these problematic behaviors. From a public health perspective, many forms of Internet use—not just gaming—continue to be recognized as potentially problematic, as evidenced by a recently-funded international research collaborative on problematic Internet use . Six studies reported both bivariable and multivariable associations; half of these showed inconsistent results. Whether consistent or inconsistent, reviews incorporated only positive findings. Six studies reported results that were not incorporated into any review; four of these had inconsistent or null findings. An additional nine studies were mentioned by reviews as reporting associations between an gaming disorder and anxiety, but using domain definitions in Fig 3 , these were not found. All but one of the three reviews that incorporated these studies reported these associations as positive. Other outcomes included the proportion of all studies within a review that measured gaming disorder with a gaming disorder-specific instrument; the proportion of reviews that specified all elements of an outcome; and the specific review- and study-level associations between gaming disorder and depression and anxiety. All reported associations within the studies were extracted from the original study reports and characterized as present and positive, present and negative, present and null, unclear, or absent. The count and type of results for each study were compared with the results reported for each study in the reviews. We also made several comparisons regarding overall conclusions about the associations between gaming disorder and depression and between gaming disorder and anxiety by comparing bivariable versus multivariable analyses, cross-sectional versus longitudinal analyses, and results from reviews classified as reliable versus results from all reviews. Current address: International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America Because of the lack of clarity around how study outcomes were selected, the reporting of outcomes that was inconsistent with study findings , the inclusion of studies that did not measure gaming disorder, and the lack of systematic assessment of bias , we determined that review conclusions were not supported by the evidence from included studies. This is further explored in the following sections. Data extracted from the reviews included information on methods for specifying eligibility criteria and outcomes, specific measurements of depression and anxiety in included studies, analyses conducted, whether and how review authors assessed risk of bias in included studies, specific measurement of gaming disorder in included studies, and all items from the AMSTAR 2 tool. Figs 5 and 6 report the positive and null associations for the depression and anxiety outcomes according to analysis type , their frequency of being incorporated into reviews, and how they are represented/reported in reviews . The shades of blue highlighting pertain to different percentages of reviews that incorporated a given relevant result from a given study . Note that only two negative associations were found and because these represent findings that were not positive and significant, they were included in the count of null findings. Overall, only the review by González-Bueso and colleagues reported any null results about depression or anxiety from any study. This study is a summary of systematic reviews of the associations between gaming disorder and depression and between gaming disorder and anxiety in any population. The review methods, including the research question, search strategy, inclusion/exclusion criteria, and risk of bias assessment, were developed a priori and described in the registered protocol ; these are also available in S1 Protocol . All data, the protocol, a list of articles excluded at the full-text screening stage with reasons for exclusion, and other supporting documentation are available on our Open Science Framework website and in Supporting Information files. In this paper, we discuss two groups of research studies: the systematic reviews and the primary studies included in those reviews . We summarize below the three conditions that had to be met for a specific measurement or scale to be classified as asssesing gaming disorder : The specific measurement or scale asked questions about computer, video, online, or digital game use in general, rather than just a single game . The specific measurement or scale asked questions about gaming or online gaming rather than Internet or computer use in general . If a study mentioned adapting a scale for video games and gave an example of an adapted question, we classified that scale as measuring gaming disorder. Otherwise, we classified the measurement according to the original scale from which it was adapted. We also conducted a sensitivity analysis to examine how our findings differed when other measurements were used with a clinical population diagnosed with gaming disorder. When the clinical population was unclear or was not diagnosed with gaming disorder and Internet addiction scales or other specific measurements/scales/interviews were used, we did not characterize this as gaming disorder . The specific measurement or scale asked questions about specific symptoms of gaming disorder rather than only experiences related to video game use in general, such as time spent playing games or the experience of time loss. Roles Data curation, Investigation, Methodology, Resources, Validation, Writing – review & editing One source of meta-bias can potentially occur when a given study included in a review reports results for a given relevant outcome in multiple ways, and the reviewer must make a choice among these to determine which result to extract for the review . In this situation, choice of the result based on the largest magnitude of treatment effect, on statistical significance, and/or on the result that supports the reviewer’s conscious or subconscious preconceptions can be problematic and lead to bias. Such bias can be preempted by completely prespecifying the five elements of an outcome . However, complete prespecification is not always possible and/or may be considered too restrictive. Moreover, choosing specific results from multiple reported analyses from multiple data sources for a given study is a multi-dimensional problem. In one study of meta-analytic methods, an examination of outcomes reported in 14 clinical trials revealed that, depending on which outcomes from the trials were chosen by the reviewers, over 34 trillion meta-analyses were possible . a = Composite reporting of outcomes in review made comparisons difficult. Is the Subject Area "Internet addiction" applicable to this article? Yes No Is the Subject Area "Depression" applicable to this article? Yes No
Affiliation Behavioral Sciences Institute, Radboud University, Nijmegen, Netherlands Click through the PLOS taxonomy to find articles in your field. In this summary of systematic reviews, we assess the reliability of current reviews that have examined the association between gaming disorder and depression or between gaming disorder and anxiety in any population. We aimed to answer the following research questions to inform directions for future research and policymaking: Do systematic reviews of the associations between gaming disorder and depression and between gaming disorder and anxiety meet reliability standards for systematic reviews? Do systematic reviews of the associations between gaming disorder and depression and between gaming disorder and anxiety distinguish between gaming disorder and other constructs, such as Internet addiction? Do systematic reviews of the associations between gaming disorder and depression and between gaming disorder and anxiety report outcomes selectively? What are the associations between gaming disorder and depression and between gaming disorder and anxiety reported in reliable systematic reviews? We conducted electronic searches of PubMed and PsycInfo for published reviews and meta-analyses . Searches combined terms related to gaming disorder and terms related to depression or anxiety . In addition, we reviewed all years of the Journal of Behavioral Addictions , including its supplements, and all proceedings of the International Conference on Behavioral Addictions. Systematic reviews are research activities that follow established, rigorous methods to summarize all relevant evidence on specific research questions that are vital for decision-making by clinicians, patients, policy-makers, and other stakeholders. The methods include framing the research question, searching for the evidence, screening studies for eligibility, assessing risk of bias and extracting data from included studies, conducting qualitative and, where merited, quantitative syntheses, and reporting the findings. Recent decades have witnessed a surge in the number of systematic reviews conducted . Multiple standards have been developed for the conduct and reporting of systematic reviews . However, research has shown that reviews in some fields provide low-quality evidence, are unreliable, and can be sources of bias themselves . Bias can sometimes be introduced due to methods used in the systematic review . Affiliation MGH Institute of Health Professions, Mass General Brigham, Boston, Massachusetts, United States of America Although all reviews acknowledged heterogeneity in measurement of problematic gaming, only one review assessed risk of bias systematically . In this context, because five studies chose to conduct qualitative syntheses instead of quantitative syntheses , we considered their results to have been combined appropriately. In one review, results were combined quantitatively despite a very high amount of statistical heterogeneity among studies . Another review classified effect sizes as small, medium, or large and presented a table of counts of effect sizes for four mental health outcomes as a way to address heterogeneity in measurement . Most reviews discussed limitations at the study, outcome, and review level, but two reviews did not discuss limitations systematically . Two investigators from among MCC, JS, and GH extracted data from each review, consulting the third investigator for resolution of discrepancies where needed. If a review did not have a summary of findings table that included the total number of studies mentioned in the results or in supplementary material, we extracted data for all studies mentioned in text or tables of the Results section. Data on depression and anxiety outcomes within each study of each review were extracted by one investigator. Extracted data for a 10% random sample of studies were validated by the second and third investigator. Citation: Colder Carras M, Shi J, Hard G, Saldanha IJ Evaluating the quality of evidence for gaming disorder: A summary of systematic reviews of associations between gaming disorder and depression or anxiety. PLoS ONE 15: e0240032. Fig 2 illustrates how we defined the domains of depression and anxiety in our study. For the outcome of depression, we restricted to scales, subscales, diagnosis, or clinical interviews for depression or more severe single symptoms related to depression, such as suicidal ideation, but excluded measurements of nonspecific symptoms, such as low energy, sleep problems, sadness, or withdrawal from social activities. For the outcome of anxiety, we included scales, subscales, diagnosis, or clinical interviews for anxiety, social anxiety, and social phobia, but excluded measurements that combined anxiety with other constructs . Assessment of AMSTAR 2 criteria showed that no study met all criteria, and some criteria were lacking in all studies. Full results can be found in S1 Data Extraction . Now that gaming disorder has been recognized as a disorder by the WHO, ensuring systematic and accurate measurement of gaming disorder in studies and accurate reporting of exposures, outcomes, and conclusions in reviews are vital for ongoing decision-making regarding diagnosis, treatment, and public health interventions. Given the established association between gaming disorder and two common mental health outcomes—depression and anxiety—we limited the scope of our study to systematic reviews that included data about these outcomes. This allowed us to explore the issue of selective outcome reporting in reviews. For the depression outcome , of the 31 studies reporting associations between gaming disorder and depression, results from 25 were included in at least one review. We found frequent under-incorporation of null results for the depression outcome by the reviews, as suggested by the paucity of blue cell highlights in the null columns. For example, the 2010 study by Rehbein and colleagues reported two findings related to depression—a positive association between gaming disorder and suicidal thoughts in one subsample, but a null association between gaming disorder and self-reported depression in the full sample. However, the three reviews that included this study and reported results for depression all reported them as positive . Affiliations Institute for Mental Health Policy Research, Centre for Addiction and Mental Health, Toronto, Ontario, Canada, School of Rehabilitation Science, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada Funding: The author received no specific funding for this work. Discover a faster, simpler path to publishing in a high-quality journal. PLOS ONE promises fair, rigorous peer review, broad scope, and wide readership – a perfect fit for your research every time. Gaming disorder or Internet gaming disorder is a disorder related to excessive video, computer, or online game play that results in psychological distress and/or functional impairment . Internet gaming disorder was included as a condition for further research in the 5 th edition of the Diagnostic and Statistical Manual and the diagnosis of gaming disorder has been added to the 11 th edition of the World Health Organization International Classification of Diseases . Gaming disorder includes symptoms related to substance use disorder, such as loss of control , functional impairment, distress, and/or interference with daily activities. The disorder is distinguished from other related disorders, such as technology overuse, Internet addiction, and social networking addiction . Recent commentaries have described gaming disorder as a clinical and public health problem in urgent need of advancements in treatment development . Elements of outcome domains required for complete outcome specification in health research. Figure adapted from ; see also the PRISMA-P statement or description of PICOS . To our knowledge, the current analysis is the first comprehensive examination of selective outcome reporting in systematic reviews of gaming disorder, a relatively new clinical entity. Due to this selective outcome reporting, incomplete outcome specification, and lack of systematic assessment of risk of bias, we found no reviews that could be considered reliable. These findings suggest that the evidence base of systematic reviews of associations between gaming disorder and the most common mental health problems must be improved. Various concerns identified during this summary of systematic reviews are worthy of further discussion. We present these in the hope that the current work drives important progress in research on gaming disorder and other types of behavioral addictions in the coming years. The searches yielded 842 records, of which, seven reviews were eligible for inclusion in this overview . The most frequent reasons for excluding articles were that they were not a systematic review or did not specify methods , did not report associations between gaming disorder and anxiety or depression , and were not specific to gaming disorder . First, the existing reviews seldom incorporated null findings or negative findings from included studies even when the studies reported such findings. This is a major concern because it seems to represent selective outcome reporting at the review level. It is vital to conduct systematic reviews and meta-analyses in ways that are replicable and consistent with best practices to ensure that all evidence is reported and that relevant studies and findings are not overlooked. Selecting which outcomes of studies to include in a review without specifying the process, which has been labelled “cherry-picking” in the clinical epidemiology literature, can lead to biased conclusions at the review level . Completely specifying all elements of outcomes or explicitly noting whether all variations of a given outcome element will be extracted is the current standard for evidence synthesis . As incomplete outcome specification may lead to trillions of potential combinations of meta-analytic results , it is inappropriate to draw meaningful and reliable conclusions about associations between gaming disorder and the common mental health problems of depression and anxiety from the reviews summarized in this paper. Selective reporting of outcomes can be hard to detect, and further research into the impact of selective inclusion of results in reviews is needed to advance the understanding of this form of bias on evidence synthesis . Because we also examined each study included in the reviews, we added an additional criterion that review authors should have: . Specified in the methods or protocol which outcomes from their eligible studies were included in the synthesis or synthesized all reported outcomes from each included study. Is the Subject Area "Internet" applicable to this article? Yes No Received: November 27, 2019; Accepted: September 18, 2020; Published: October 26, 2020 We adapted the definition of “reliability” of systematic reviews developed by Cochrane Eyes and Vision . This definition, in turn, was informed by items identified from the Critical Appraisal Skills Programme , A Measurement Tool to Assess systematic Reviews , and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses tools . According to this definition, a review is reliable when its authors did each of the following: . Defined eligibility criteria for including studies; . Conducted a comprehensive literature search for studies ; biggest video game news stories