Home

TSA user manual - Copenhagen Trial Unit

Contents

1. Figure 13 The TSA starting window 4 1 1 Creating a new meta analysis To create a new meta analysis go to the menu bar and select File gt New Meta analysis A dialogue box will appear figure 14 allowing you to name your meta analysis choose the type of data that will be meta analysed dichotomous data or continuous data define which two interventions are being compared define whether the outcome type is negative or positive and add comments Press Create to create the new meta analysis Press Cancel to cancel this action If you want to edit the name of the meta analysis the interventions or your comments go to the menu bar and select File gt Edit meta analysis The dialogue box shown in figure 14 should then re appear 58 User Manual for TSA Copenhagen Trial Unit 2011 New Meta analysis x Outome A Data Type Dichotomous Name Comparison Label for Group 1 Label for group Outcome type it Negative Positive Comments f Figure 14 Dialogue box for creating a new meta analysis Note binary data negative outcomes are outcomes like mortality stroke or new cancer incidences positive outcomes are outcomes like survival clearance of a virus or smoking cessation For continuous data negative outcomes are outcomes where an increase in the mean response is a bad thing e g increase in depression score and positive outcomes are outcomes where an increase in the mean
2. User Manual for TSA Copenhagen Trial Unit 2011 We performed TSA on these data We calculated the information size required to demonstrate or reject a 20 relative benefit increment smoking cessation being the outcome of benefit We assumed a 14 event proportion in the control group which was roughly the median and average control group event proportion We used a type error of 5 and a type II error of 20 We did not correct for heterogeneity With these settings we calculated the required information size to 5218 patients As the number of patients included in the meta analysis did not exceed the required information size we also applied futility boundaries to potentially facilitate a firm negative conclusion Cumulative Z acore TSA 5218 8 hy Favors Intensity 3 in SI ec Number of i a patients e A EEE A E E EEE AEP E EAEE E E AE E EEA E E E E TT Te Linear scaled 2 pee j ee sA m 4 _ a i e TD i D E pa far 5 Pa fou g Pi aS a u Bs a a Figure 56 The required information size to demonstrate or reject a 20 relative increase in benefit on smoking cessation with a control group proportion of 14 an alpha of 5 anda beta of 20 is 5218 patients vertical red line The red dashed lines represent the trial sequential monitoring boundaries and the futility boundaries The solid blue line is the Cumulative Z curve The resulting trial
3. 66 User Manual for TSA Copenhagen Trial Unit 2011 The trial data appears in the same area where you type in new trial data and you can now edit data This area will now contain an Edit Trial button instead of an Add Trial button To edit the trial data change the content in the fields you want to edit and click on the Edit Trial button If you wish to delete a trial select the row for the trial you wish to delete and press the Delete Trial button in the Edit Delete Selected Trial area Alternatively you can select the row for the trial you wish to delete and press the lt Delete gt button on your keyboard 4 3 Defining your meta analysis settings The TSA program provides a number of options for performing meta analysis You can choose between a number of effect measures statistical models zero event data handling methods for dichotomous data and confidence interval coverage levels All of these options can be set in the Meta analysis tab to the left of the Trials tab figure 24 Figure 24 Click on the Meta analysis tab when you want to set your effect measure statistical model or zero event handling method In the left side of the window you will find the Set Effect Measure and Model area the Set Zero Event Handling area and the Set Confidence Intervals area figures 25 28 In the middle of the window you will find the Meta analysis Summary area 4 3 1 Choosing your association measure The TSA program
4. B E CHIVE pa 71 96 P 0 05 Number of 2000 4000 patients Patients included Required information size Figure 9 Example of a meta analysis that becomes conclusive according to the O Brien Fleming boundaries after the fifth cumulative significance testing In the above examples figure 7 9 the monitoring boundaries are constructed only for the positive half of the y axis Two sided symmetrical significance testing boundaries can be constructed on both the negative and positive half of the y axis The TSA program allows for both one and two sided significance testing When the outcome measure for binary data meta analysis is defined as a failure e g death or relapse Z values on the upper half of the y axis will indicate benefit of the experimental intervention whereas Z values on the lower half will indicate harm The monitoring boundaries values for the Z curve are a function of the alpha spending function they are calculated by numerical recursive integration according to Reboussin et al Though all boundary values are discrete points calculated for each cumulative update of the meta analysis the TSA program connects these points and creates one continuous boundary line for better visual interpretation 2 2 5 Adjusted confidence intervals following trial sequential analysis Just as repeated significance tests affects the overall type error it also affects the construction of confidence intervals For example whe
5. Europ Hearj J Mayo Clin Proc 2008 J Alim Pharm amp Ther 2008 Europ Heart J 2008 BMJ 2007 Pentoxifylline vs control for alcoholic hepatitis i Off pump vs on pump CABG for atrial fibrilation ii Off pump vs on pump CABG for myocardial infarction i Perioperative insulin infusion vs control for Mortality li Perioperative insulin infusion vs control for Morbidity Glucocorticosteroids vs control for alcoholic hepatitis Prophylactic steroid use vs control for patients undergoing cardiopulmonary bypass Antithrombin IIl vs control for reducing cardiac Table 4 Overview of published meta analyses where trial sequential analysis was applied 101 User Manual for TSA Copenhagen Trial Unit 2011 6 Appendixes 6 1 Effect measures for dichotomous and continuous data meta analysis The standard errors of the respective effect measures are calculated similarly to the methods used in Review Manager v 5 For each trial we denote the number of observed events e g deaths in the two intervention groups ea and eg and the total number of participants na and neg in the two intervention groups The standard errors for risk differences relative risks and odds ratios are calculated using the following formulas se RD AR A Np ae ea fr l e l e For a Peto s odds ratio the standard error is given by se OR V1 v where n n n n 1 102 User
6. Bangalore S Wetterslev J Pranesh S Sawhney S Gluud C Messerli FH Perioperative beta blockers in patients having non cardiac surgery a meta analysis Lancet 2008 372 1962 1976 Devereaux PJ Beattie WS Choi PT et al How strong is the evidence for the use of perioperative beta blockers in non cardiac surgery Systematic review and meta analysis of randomised controlled trials BMJ 2005 331 313 321 DeMets D Lan KK Interim analysis the alpha spending function approach Statistics in Medicine 1994 12 1341 1352 111 User Manual for TSA Copenhagen Trial Unit 2011 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 Lan KK DeMets D Discrete sequential monitoring boundaries for clinical trials Biometrika 1983 659 663 Pocock S Interrim analyses for randomized clinical trials the group sequential approach Biometrics 1982 38 153 162 GRADE Working Group Grading quality of evidence and strength of recommendations in clinical practice guidelines Part 1 of 3 An overview of the GRADE approach and grading quality of evidence about interventions Allergy 2009 64 669 677 Guyatt G Mills E In the era of systematic reviews does the size of an individual trial still matter PLoS Medicine 2008 5 e4 Sutton AJ Cooper NJ Jones DR Evidence synthesis as the key to more coherent end efficient research BMC Medical Research Methodol
7. CAS vs carotid endarterectomy on i death myocardial infarction or stroke li periprocedural death or stroke iii periprocedural stroke i Angiotensin receptor blockers vs comparison effect on cancer risk and on cancer related death li Angiotensin converting enzyme inhibitors vs com parison effect on cancer risk and on cancer related death iii Beta blockers vs comparison effect on cancer risk and on cancer related death iv Calcium channel blockers vs comparison effect on cancer risk and on cancer related death v Diuretics vs comparison effect on cancer risk and on cancer related death i Inhaled nitric oxide vs control for acute respiratory distress syndrome li Inhaled nitric oxide vs control for lung injury Peginterferon alfa 2a vs peginterferon alfa 2b for hepatitis C Ribavirin plus interferon vs interferon for hepatitis C Hypothermia vs control after cardiac arrest i Probiotics vs control to reduce mortality in newborn li Probiotics vs control to reduce necrotizing entercolitis in newborn Salivary cortisol in depressed patients vs control persons i Perioperative beta blockade vs placebo for mortality i Perioperative beta blockade vs placebo for myocardial infarction Ribavirin monotherapy vs placebo for hepatitis C 100 User Manual for TSA Copenhagen Trial Unit 2011 Whitfield K Moller CH Ghandy GY Rambaldi A Whitlock R Afshari A The Cochrane Library 2009
8. O Layout settings i o i Print current Graph Generate TSA Report Figure 48 The Graph window Tests and boundaries Layout Conventional 2 sided LIL So Figure 49 The significance tests and Z curve listings area In the Set Graph Layout area you will find a number of options for changing the general graph presentation If you click on the Layout settings button a pop up window will appear figure 50 providing you with the options of adjusting the width of the x axis and y axis the coordinate font size or the font and the size of the fixed text components on the graph 84 User Manual for TSA Copenhagen Trial Unit 2011 Graph Layout Settings 7 x Fixed text Axes Font Serif Line width Font size 12 Font size OK Cancel Figure 50 General graph layout settings to adjust fixed text components font and font size the width of the x axis and y axis and the coordinate font size On the information axis the distance between boundaries and the distance between the Z values are conventionally displayed with respect to the relative increase in information The TSA program automatically displays these distances in this scaled manner Set Graph Layout Trial Distance Scaled Layout Equal Figure 51 Select Equal distance for equal distance between trials on the x axis In some instances however other layouts may provide a
9. i i k where 7 denotes the between trial variance One advantageous property of the diversity measure D is that the above derivations are generalisable to any given meta analysis model Thus if we wish to meta analyse some trials using an alternative random effects model with total variance vp the diversity measure and the corresponding adjustment factor simply take the expression Vp V V Paoa and AF VR Ve Estimates of variability and in particular between trial variability may be subject to both random error and bias For this reason in some situations using D or If based on the available data may be inappropriate In meta analyses that only include a limited number of trials e g less than 10 trials estimates of heterogeneity and the between irial variance may be just as unreliable as intervention effect estimates from small randomised clinical trials e g trials including less than 100 patients When a meta analysis is subject to time lag bias i e when trials mostly with positive findings have been published the between trial variance will typically be underestimated This underestimation occurs because the early set of included trials are likely to have yielded similar positive intervention effect estimates Later meta Ros User Manual for TSA Copenhagen Trial Unit 2011 analyses updates are likely to include more trials with neutral or even negative findings
10. information size This information size is analogous to the required sample size in a single randomised clinical trial 9 Controlling the risk of type error involves an alteration in the way we measure statistical significance If a meta analysis is subjected to significance testing before it has surpassed its required information size the threshold for Statistical significance can be adjusted to account for the elevated risk of random error 746 111223 Alternatively the test statistic itself can be penalised in congruence with the strength of the available evidence TSA provides the option to use both of these approaches to control the type 1 error Controlling the risk of type Il error before a meta analysis surpasses its required information size involves setting up thresholds rules for when the experimental intervention can be deemed non superior and or non inferior to the control intervention Osc User Manual for TSA Copenhagen Trial Unit 2011 The methods for adjusting significance thresholds i e controlling the type error build on methods introduced by Armitage and Pocock these methods are referred to as group sequential analysis In Armitage s and Pocock s group sequential analysis it is necessary to know the approximate number of patients randomised between each interim look at the data In randomised clinical trials interim looks on accumulating data are typically pre planned and i
11. lt a a aE Pr Z gt q Z lt q and and Z lt c lt o a UR aR 39 User Manual for TSA Copenhagen Trial Unit 2011 The actual a spending function used can be any monotonically increasing function One well known example is a t t a When all significance tests are performed at an equal distance with respect to the information fraction scale this a spending function will yield equal thresholds for the Z values 1 C7 Co Cx This adjustment was first proposed by Pocock A more general a spending approach is the power family a spending function defined as a t a Power family a spending functions where p gt 7 and where all significance tests are performed at equal distance will yield more conservative thresholds for early significance tests than for later significance tests In general the thresholds for absolute values of the Z curve will be monotonically decreasing when the a spending function is convex and all significance tests are performed at equal distance Monotonically decreasing thresholds which result from the monotonically increasing functions are desirably because the impact of random error is typically inversely proportional to the amount of accumulated information Although an infinite combination of decreasing thresholds exists some sets of thresholds may be preferable From advanced probability theory the a spending function that yield theoreticall
12. no effect We approximated that about 3800 patients 1900 patients in each intervention group would be required to yield a conclusive positive meta analysis Figure 63 About 4000 patients 2000 patients in each intervention group would be required to yield a conclusive meta analysis showing futility Figure 64 5 6 Other published trial sequential analysis applications The authors of this manual have authored several systematic reviews for which trial sequential analysis was applied to at least one meta analysis 147493 65 74 Table 4 provides a brief overview of these publications ordered by year of publication 99 User Manual for TSA Copenhagen Trial Unit 2011 First author T Bangalore 7 Bangalore TT Bangalore Afshari A Awad T Brok J Nielsen N Tarnow Mordi Wo Knorr U Bangalore S Brok J Journal year BMJ 2011 Archives of Neurology 2011 Lancet Oncology 2011 The Cochrane Library 2010 Hepatology 2010 J Alim Pharm amp Ther 2010 Int J Cardiol 2010 Pediatrics 2010 Psychoneuroendo crinology 2010 The Lancet 2009 The Cochrane Library 2009 Meta analyses Angiotensin receptor blockers ARB vs control for i non fatal myocardial infarction ii all cause mortality iii cardiovascular mortality iv angina pectoris v stroke vi heart failure vii new onset diabetes Carotid artery stenting
13. oh Ny O Brien Fleming boundaries T K3 Z 1 96 P 0 03 Number of 2000 4000 patients Patients included Required information size Figure 8 Example of a meta analysis including a false positive Z value at the fifth cumulative significance testing In the example given in figure 8 the required information size is again 4000 patients and the obtained information is now 2000 patients The final Z value is smaller than 1 96 this result would have been inconclusive using either conventional or boundary techniques There are however preceding Z values that had been calculated in the cumulative process including one with a value greater than 1 96 This example illustrates how a cumulative Z curve could cross the conventional threshold for significance in an early meta analysis only to be declared not significant in a later meta analysis O Brien Fleming boundaries can prevent such premature false positive conclusions In the example given in figure 9 the required information size and the attained information size are the same as those in figure 8 Here the Z value calculated at the fifth significance test is extreme enough the Z curve crosses the O Brien Fleming boundaries and the meta analysis can be declared as conclusive with regard to the anticipated intervention effect leading to the required information size _ 43 User Manual for TSA Copenhagen Trial Unit 2011 S Ny O Brien Fleming boundaries 2 E
14. 1 Letting Srixeq denote the required information size for a fixed effect meta analysis given by equation 1 vr denote the total variance in the random effects model meta analysis and ve denote the total variance in the fixed effect model meta analysis the heterogeneity adjusted information size can be derived using the following formula IS Random vV es R loing Vir Given that the anticipated intervention effects in the fixed F and random effects r models are approximately equal that is given r of it can be shown mathematically that in the special case where all trials in a meta analysis are given the same weights the heterogeneity adjustment factor AF takes the form Where I is the inconsistency factor commonly used to measure heterogeneity in a meta analysis It is important to remember that in any case where the trial weights are not equal using will lead to an underestimation of the adjustment factor and thus an underestimation of the required information size In this situation we can define a measure of diversity D as the quantity compelled to satisfy the equation oy User Manual for TSA Copenhagen Trial Unit 2011 where w denotes the trial weights in the fixed effect model and w denotes the trial weights in the random effects model Solving the equation with respect to D we get k Xw Dw T N l a a D 8 F 4 k Oooo kpo v 2v gt
15. 9 Test Tor owarall efie Z 2 59 P 0 003 Total 965 CI 1759 1797 Ea Tolal events 207 iOf pumpl 470 On gume Test for haberagenety Chi 42 52 df 26 F 0 02 F 33 9 Test for overall atte Z 4 27 P 0 0007 DiI O2 gs 1 z 5 10 Favours ol fume Favours onm pump Figure 57 Forest plot of the effect of off pump vs on pump CABG on atrial fibrillation In the meta analysis of trials with low risk of bias 1050 patients the effect was not significant 0 63 0 35 to 1 13 the estimated heterogeneity was I 77 and the estimated diversity was D 79 0 Trial sequential analysis of atrial fibrillation We calculated two required information sizes for this example First we calculated the information size required to demonstrate or reject an a priori anticipated intervention effect of a 20 relative risk reduction alpha of 1 9 User Manual for TSA Copenhagen Trial Unit 2011 and beta of 10 which was 7150 patients The value of 20 anticipated intervention effect was chosen because it was believed to represent a reasonable intervention effect in this clinical situation Second we calculated the information size for the meta analysed estimate of the relative risk reduction from the low bias risk trials included in the review 36 9 which was 1964 patients Cumulate 4 cOre Required Information Size 7150 Number of patients Figure 58 The heterogeneity adjusted required information size to d
16. 96 Z lt 1 1 Pr 0 05 Pr Z lt 1 96 Z lt 1 96 Pr Z lt 1 96 or Z lt 1 96 gt 0 05 Z lt 1 96 Pr Z lt 1 96 Z lt 1 96 Pr Z lt 1 96 Pr Z lt 1 96 Z lt 1 96 1 Pr Z lt 1 96 Pr Z lt 1 96 Z lt 1 96 Pr Z lt 1 96 or Z lt 1 96 Where the inequality is apparent from the fact that Pr Z lt 1 96 or Z lt 1 96 gt Pr Z lt 1 96 Z lt 1 96 The above is easily generalisable for any value of a and for any number of repeated significance tests 6 3 2 Alternative methods not implemented in the TSA software A wide range of methods are available for repeated significance testing in randomised clinical trials some of which may also find application in meta analysis The approaches implemented in the TSA software are all approaches constructed around monitoring of the standardized Z statistic or at least an adjustment hereof Other sequential approaches which have received some attention in the context of meta analysis are constructed to monitor other statistics One approach that has recently received some attention is the sequential analysis monitoring of efficient scores or the likelihood score statistic for the meta analysed effect In the standard meta analysis setting the efficient score for each trial is simply the estimated trial treatment effect multiplied by its variance and the efficient score for a meta analysis is the sum of trial effi
17. Caputo 2002 0720 o 20 2002 fanmvar 2002 0 30 0 30 2002 Camer 2003 1 32 1 33 2 16 2003 Raja 2003 77150 67150 id 11 2003 Awan 2004 5 35 5 35 12 19 200 Gerola 2004 6 80 3 80 6 80 2004 Khan 2004 1 54 0 49 1 59 2004 Legare 2004 4 150 1 150 3 38 2004 PRAGUE 4 2004 4 208 3 192 7 29 2008 Sevanayagam 2004 2 30 2 30 4 48 2004 Kobayashi 2005 2 81 2 86 A 8 2005 Ascione 2006 1 20 0 20 1 62 2006 Michaux 2006 1 25 4 25 3 57 2006 dares 2007 0 10 0 10 2007 Kunes 2007 O 17 07 17 2007 Orkara 2007 O 22 of22 2007 Subtotal 95 Cl 1025 1010 65 75 Total events 35 Of pump 28 On pump Test for heterogeneity Chit 4 58 df 11 F 0 95 F 0 Test for overall affect Z 0 80 P 0 43 03 Trials with unclear randomisation and or lack of blinding Vural 1995 07 25 0 25 1995 Czemy 2000 07 15 0 15 2000 Kochamba 2000 1 25 1 29 2 16 2000 Wandschneider 2000 0 52 2 6T 1 77 2000 Baker 2001 17 132 o 14 1 66 2001 Gzemy 2001 0 40 0 40 2001 Guler 2001 0 40 1 148 1 61 2001 Farmari 2003 07 11 07 14 2003 Sahiman 2003 o 24 3 26 1 89 2003 Vedin 2003 1 33 0 37 1 60 2003 Volissans 2003 07 27 0 27 2003 Synnergren 2004 07 2 0 26 2004 Blacher 2005 0 13 0 15 2005 Cavala 2006 1725 07 25 1 61 2006 Malik 2006 07 25 07 25 2006 Nesher 2006 0 60 0 60 2006 Quaniers 2006 0 40 o 40 2006 Rachwalik 2006 0 21 0 21 2006 Subtotal 95 Cl 514 52d 12 30 Total events 4 Off pump 7 On pump Test for heterogeneity Chi 5 17 df 6 P 0 52 F 0 Test for overall affe
18. OO aE gO OR E SO Oe 55 323 1 WANY doesnt TSA Stari arios a E a 56 3 4 Starting RMS CONVENer sra iea E AE A 57 OAV Why doesh t RMS stalt iann E T T A 57 d How tome TOA na a a a awe pond acace susesavaaeen oui ca acta 58 Jl Getting Stane a rca A a A 58 4 1 1 Creating a new Meta ANALYSIS 0 0 0 ec cessecceessssceeessseeeceessseeceessseeceesssseceeesseeeeeesseeeeeeas 58 4 1 2 Saving a TSA file and opening an existing TSA fil cece ccccessecceeesseeeeeeeteeees 60 4 1 3 Importing meta analysis data from Review Manager V 5 ccccccceessecceessseceeeenseees 60 4 2 Adding editing and deleting trials cccssssssececeessssccecccesssaeeecesssssaeeecceestsaaeeees 64 42 t PNOGIIG Mals seinan e uisin bem O EE davsassueslacietes 64 4 2 2 Editing and deleting WIAlS Kx ccssihidintatwartrneGnuetederanBaliGetatwaledatataGecastetes 66 4 3 Defining your meta analysis settings sssessesseseesseseeesssesssssrrsssesersserresssrressssressserrrssen 67 4 3 1 Choosing your association Measure ccc ccesecceesssccceeesceeeeessseeecessseeeeeesseeeeestseeeeees 67 4 3 2 Choosing your statistical model 00sneoeeeeseeeesseeessssssssssrersssrerssseressssressssrresssrreessrre 68 4 3 3 Choosing a method for handling Zero event data eee cccccccccessseeceesseeeeeessseeees 68 4 3 4 Choosing the type of Confidence interval cee cceessscceessseeeceessseeeesssseeeeessseeeeens 69 4 4 Applying adjusted significan
19. Tools menu select Folder Options and go to the File Types tab find the JAR extension in the list click Change select Java TM Platform SE binary from the list and click OK If Java TM Platform SE binary is not in the list click Browse and locate the javaw exe in the JRE s bin folder Its default path is C Program Files Java jre6 bin If your operating system is not Microsoft Windows please consult the user manual for your operating system 3 4 Starting RM5 Converter To start the Review Manager 5 data converter program presently however for dichotomous outcomes only double click the RM5Converter jar file 3 4 1 Why doesn t RM5 start The RM5Converter jar has the same basic prerequisites as TSA jar so if you are having difficulty opening it please consult section 3 3 1 Also RM5 Converter jar requires the TSA jar file to be able to run For this reason TSA jar has to be located in the same folder as RM5Converter jar Secs User Manual for TSA Copenhagen Trial Unit 2011 4 How to use TSA 4 1 Getting started When TSA is started a window similar to figure 13 should appear The starting window should contain a menu bar with the menus File Settings and Help as well as five greyed out non selectable tabs Meta analysis Trials TSA Graphs and Diversity TSA Trial Sequential Analysis Viewer version 0 9 Beta 0 x No Meta analysis Defined
20. beta of 20 is 5942 patients vertical red line To the left the red inward sloping dashed lines make up the trial sequential monitoring boundaries To the right the red outward sloping dashed lines make up the futility region The solid blue line is the cumulative Z curve We calculated the information size required to demonstrate or reject an a priori anticipated intervention effect of a 33 relative risk reduction The value of 33 was chosen because it was believed to represent a reasonable intervention effect in this clinical situation In contrast to the information size calculation for atrial fibrillation we used a maximum type error of 5 and a maximum type Il error of 20 80 power We used the median proportion of myocardial infarctions in the on pump groups excluding the zero event trials as our control group event proportion 3 9 Collectively these 96 User Manual for TSA Copenhagen Trial Unit 2011 assumptions yielded a required information size of 5942 The cumulative Z curve crossed the futility boundaries Figure 61 and we are therefore able to infer that neither off pump CABG nor on pump CABG is more than 33 more effective than the other This finding of course comes with a 20 risk of being a false futile finding the type II error is 20 5 5 Estimating the sample size of a new clinical trial When a meta analysis has neither crossed the monitoring boundaries nor the futility bound
21. better basis for visual interpretation The TSA program also provides the layout format used in the paper by Pogue and Yusuf which displays trials at equal distance on the information axis and displays the trial titles at a 45 angle below the x axis To choose this layout format click on the Trial Distance drop down box in the Set Graph Layout area and select Equal figure 51 a gt User Manual for TSA Copenhagen Trial Unit 2011 Cumulative 2 acOre O TSA 2546 g n 7 6 Favors Expermental Intervention bo we b bJ a fai Mi fh be amp he ate ay a g rl ae ae Ki ai os st Pe a P g P gy aT e G P m z 3 F 4 pE gj Bine o E E 5 D 7 F _ bm Figure 52 Select information axis scaling display format Adjusted significance tests based on a spending functions are in effect adjusted thresholds for the Z curve whereas adjusted significance tests based on the law of the iterated logarithm penalties are in effect adjusted test statistics that should be interpreted in relation to single test significance test thresholds Thus combining these two approaches in one graph is not meaningful The TSA program provides separate graphs for adjusted significance tests based on a spending functions and the law of the iterated logarithm penalties To see the graphical representation of the calculated a spending boundaries select the Adjusted Boundaries tab a
22. estimating an overall pooled estimated treatment effect would be more appropriate In this case the over all meta analysis would not be performed In this example one could use the average of the two 60 20 2 40 for the primary information size calculation but acknowledge that the required information size may be as large as the one based on 60 heterogeneity adjustment or as low as the one based on 20 heterogeneity adjustment As another example one could conceive and 5902 User Manual for TSA Copenhagen Trial Unit 2011 construct a number of best and worst case scenarios whatever those might be by adding imaginary future trials to the current meta analysis This approach would allow one to assess the robustness and reliability of the D estimate and construct a spectrum of realistic or acceptable degrees of heterogeneity which could readily be utilized for sensitivity analysis Estimating the control group event proportion and an anticipated intervention effect The estimation of the control group event proportion and an anticipated intervention effect are important determinants of the calculated required information size when doing TSA Every effort should therefore be made to make these estimates as accurate and realistic as possible For binary data control group event proportion can be estimated by using clinical experience and evidence from related areas An a priori estimate of a realistic interve
23. for heterogeneity The choice of random effects model should involve a sensitivity analysis comparing each approach If the DL SJ and BT approaches all yield similar statistical inferences i e point estimates and confidence intervals it would be reasonable to use the DL approach and have confidence that the estimation of between trial variance is reliable lf two or all of the three approaches differ one should carry out meta analysis with both or all approaches and consider the results according to the underlying properties of each approach For example if the DL and SJ approaches produce different results two possible explanations should be considered 1 the meta analysis is subject to moderate or substantial heterogeneity and the DL estimator therefore underestimates the between trial variance and yields artificially narrow confidence interval and 2 the meta analysis is subject to mild heterogeneity and the SJ estimator therefore overestimates the between trial variance and yields artificially wide confidence intervals In this situation one should then carry out meta analyses with the two approaches and consider the implications of each of the two scenarios being true 19 User Manual for TSA Copenhagen Trial Unit 2011 2 1 4 Methods for handling zero event trials In dichotomous trials the outcome of interest may be rare For example the occurrence of heart disease from the use of hormone replacement ther
24. of events and patients in this case less than 100 events and less than 300 patients should not be trusted Point estimates from this meta analysis did not appear to be sufficiently reliable until at least about one hundred events and one thousand patients were accrued Adjusted confidence intervals serve to guard against spurious inferences at early stages of a meta analysis and appropriately converge to resemble conventional confidence intervals as the accrued number of patients approaches the required information size 94 User Manual for TSA Copenhagen Trial Unit 2011 Review Off pump Versus on pump coronary artery bypass grafting for ischaemic heart disaasa Comparison 10 Off pump Versus on pump coronary artery bypass surgery Shor tearm follow up Outcome 03 Myocardial infarction study Off pump Or pump RR random Weight or sub category ni miN 95 Cl w Yoar 01 Trials with adequate randomisation and blinded outcome assessmant BHACAS 1999 1 100 4 100 3 40 1999 OCTOPUS 2001 7 142 6 139 14 15 2001 BHACAS I 2002 0 100 1 101 1 58 2002 SMART 2003 1 100 2 100 2 82 2003 Subtotal 95 Cl 442 440 21 95 Total events 9 Off pump 13 On pump Test for heterogeneity Che 1 96 df 3 P 0 58 F 0 Test for overall affect Z 0 68 P 0 50 02 Trials with adequate randomisation but lack blinding Diegeler 2000 o 20 0 20 2000 Gulielmos 2000 0720 07 20 2000 Matata 2000 0 10 0 10 2000 Panttila 2001 irii ifii 2 30 2001
25. of repeated significance tests see section 2 2 3 In the considered meta analysis data sets there were some years when more than one trial was published For these years we have analysed trials in alphabetical order according to the last name of the first author 5 3 1 Confirming the answer is in To illustrate the application of TSA for asserting the answer is in we used the outcome of atrial fibrillation in this on pump vs off pump meta analysis Occurrence of atrial fibrillation was reported in 30 trials including 3634 patients with two zero event trials According to conventional standards for significance testing off pump CABG was significantly superior to on pump CABG in reducing atrial fibrillation RR 0 69 95 CI 0 57 to 0 83 Figure 57 The estimated inconsistency was I 47 3 and the estimated diversity was D 49 0 90 User Manual for TSA Copenhagen Trial Unit 2011 Fev iew Of PUMP vecus or pump Coronary artery bypass grafling for iechaemic heart disease Compaenson 10 Olf pump versus OF PUMp coronary anery Oy pass surgery short lenm follow up Ouicome 05 Atrial tforiiation Siudy Of pump On suma RR random Weight OF Sub calegory nN mH JET Ci a Year OT Trials with adequele randomisation and blinded ouicoma assessment SHACAS 1239 11 100 ata 1 33 1555 OCTOPUS 2001 707142 23 33 6 327 2001 JHATAS II 2072 147100 23 101 H15 2o02 SMAAT 7003 16 101 22 100 s 08 2003 Sunt
26. option of ignoring one or more added 65 User Manual for TSA Copenhagen Trial Unit 2011 trial s for when you are performing your meta analyses Simply check the Ignore check box to ignore a trial The fourth column should display the trial data For dichotomous trials the format is Intervention Events Total Control Events Total For continuous data the format is Intervention Mean Reponse Standard Deviation Group Size Control Mean Reponse Standard Deviation Group Size TSA Trial Sequential Analysis Viewer version 0 9 Beta 7 10 x File Meta analysis Trials TSA Graphs Diversity Add Dichotomous Trial Study Year Event Total Intervention Control Low Bias Risk Comment Add Trial Edit Delete Trial Edit Selected Delete Selected lgnore Trials Low Bias Risk trials High Bias Risk trials All None 2009 Test Trial 1 2010 Test trial 2 Low _ Figure 22 List of added trials marked within the red ellipse 4 2 2 Editing and deleting trials To edit trial data first select the row for the trial you wish to edit and then click on the Edit Trial button in the Edit Delete Selected Trial area figure 23 Alternatively you can double click on the row for the trial you wish to edit Edit Delete Trial Edit Selected Delete Selected Figure 23 The Edit Delete Selected Trial area
27. provides the same effect measures as Review Manager version 5 see section 2 1 1 for a description of these measures To select an effect measure first click on the Effect Measure drop box in the Set Effect Measure and Model area in order to display the available effect measures figure 25 marked area in the left side picture then click on the effect measure you wish to use for your meta analysis 67 User Manual for TSA Copenhagen Trial Unit 2011 Set Effect Measure and Model Effect Measure Model Odds Ratio Peto Odds Ratio Figure 25 Select effect measure by clicking on the Effect Measure drop box 4 3 2 Choosing your statistical model The TSA program provides four statistical models for pooling meta analysis data three of which are variants of the random effects model see section 2 1 2 To set your statistical model first click on the Model drop box to display the available effect measures figure 26 marked area in the left side picture and then click on the model you wish to use for your meta analysis Set Effect Measure and Model Set Effect Measure and Model Effect Measure Relative Risk r Fixed Effect Model Fixed Effect Model Random etfects DL Random effects SJ Random effects BT Model Figure 26 Select effect measure by clicking on the Model drop box 4 3 3 Choosing a method for handling zero event data The TSA program provides three methods
28. remaining theoretical sections on repeated significance testing sections 2 2 2 to 2 2 5 we will assume that all statistical tests are two sided We will also assume that all test statistic values Z are absolute values We assume the latter because the involved algebra becomes much simpler by doing so For example in defining two sided thresholds for a non absolute test statistic one would need to consider the probability that Pr Zs c or Zzc rather than Pr Z c Problems with repeated significance testing Conventional single significance tests can be considered reliable if enough data has accumulated In meta analysis a single significance test can be considered reliable once the required information size is surpassed 4 6 11 12 20 99 If we perform a single test for statistical significance at or after a meta analysis has surpassed its required information size statistical significance testing simply entails determining an appropriate threshold c that will make equations 2 and 3 true For example for a 5 we would consider c 1 96 appropriate if the meta analysis data had not previously been subjected to significance testing When a cumulative meta analysis is subjected to significance testing more than once before surpassing its required information size the situation becomes more complex Consider the example where a meta analysis is updated once and where the conventional 5 maximum type error is use
29. sequential analysis is shown in figure 56 After the first and second trial the cumulative Z statistic crossed above 1 96 which corresponds to the nominal threshold for statistical significance using conventional techniques From the third trial onwards the meta analysis was no longer nominally statistically significant When the last trial was added the meta analysis crossed below the futility boundaries demonstrating with 80 power 89 User Manual for TSA Copenhagen Trial Unit 2011 that the effect of an intensity 3 intervention is not larger than a 20 relative increase in smoking cessation That is within the set assumptions for confidence and effect size this intervention is ineffective 5 3 Confirming a positive result To illustrate the application of TSA for asserting positive results we used data from a systematic review comparing off pump and on pump coronary artery bypass grafting surgery CABG For this example the adjusted significance boundaries for the cumulative Z curve were constructed under the assumption that significance testing may have been performed each time a new trial was added to the meta analysis Given the considerable amount of attention that has been given to the off pump vs on pump debate over the last decade this assumption seemed reasonable We used the monitoring boundaries based on the O Brien Fleming type alpha spending function which are relatively insensitive to the number
30. statistically significant and which results are not 1 46 11 12 14 15 24 25 Alternatively one can penalise the test statistic according to the strength of evidence and the number of performed significance tests the law of the iterated logarithm The TSA software provides methods for both approaches each building on theorems from advanced probability theory The first approach uses methodology developed for repeated significance testing in randomised clinical trials i e statistical monitoring boundaries The second approach penalizes that is decreases the test statistic according to the strength of information available in the meta analysis and the number of performed significance tests Test A Test B statistic statistic Significant aoahn ee mee Non significant a ee his vene Figure 4 Examples of significance threshold adjustment stipulated monitoring boundaries A and penalised test statistic stipulated B to avoid false positive statistical test results in two cumulative meta analyses A and B Figure 4 A illustrates an example of a meta analysis scenario where a false positive result is avoided by adjusting the threshold for statistical significance by employing monitoring boundaries Figure 4 B illustrates an example where a false positive result is avoided by appropriately penalizing the test statistic SRo User Manual for TSA Copenhagen Trial Unit 201
31. testing outlined in section 2 2 3 is to penalise the Z values according to the strength of the available evidence and the number of repeated significance tests In advanced probability there exists a theorem the law of the iterated logarithms which tells us that if we take a standard normally distributed variable such as a Z value and divide it by the logarithm of the logarithm of the number of observations in the data there will be a 100 probability that this fraction will assume a value between 2 and V2 In the context of Statistical testing this law can be utilised to control exaggeration of type 1 error in meta analysis due to repeated significance testing Dividing a standard normally distributed test statistic by the logarithm of the logarithm of the information available provided enough data has accumulated can provide good control of the behaviour of the employed statistical test Lan et al applied this theory introducing a penalty for the Z values obtained at each significance test and creating adjusted penalised Z values Z given by where Z is the conventional Z value l is the cumulative statistical information at the j th significance test see section 2 2 1 under alternatives to ATs User Manual for TSA Copenhagen Trial Unit 2011 accumulating number of patients and is some constant that will ensure good control of the maximum type error Lan et al used simulation to estimate proper c
32. the th trial For dichotomous data meta analysis Y will either be the estimated risk difference the log relative risk the log odds ratio or the log of Peto s odds ratio for the i th trial For continuous data meta analysis Yi will be the estimated mean difference for the i th trial Let u be the true effect of the i th trial and the let u be the true underlying intervention effect for the entire meta analysis population Let a denote the variance sampling error of the observed intervention effect in the th trial ee User Manual for TSA Copenhagen Trial Unit 2011 In the fixed effect model the characteristics of the included trials patient inclusion and exclusion criteria administered variants of the intervention study design methodological quality length of follow up etc are assumed to be similar This is formulated mathematically as u1 u2 uk u The observed intervention effects of the individual trials are then assumed to satisfy the distributional relationship Y N u of The weight of a trial wi is defined as the reciprocal of the trial variance and hence the trial weights in a fixed effect model become w o The pooled intervention effect i is obtained as a weighted average of the observed intervention effects of the individual trials and has variance Var Sa In the random effects model the intervention effects are assumed to vary across trials but with an underlying
33. this problem involve the use of more complex methodology to adjust the uncertainty associated with estimating the between trial variation One option is to use the random effects approach by Biggerstaff Tweedie which incorporates the uncertainty associated with estimating between trial variance when using the conventional DerSimonian Laird estimator see section 2 1 3 Another option is to apply Bayesian meta analysis where a prior distribution is elicited for the between trial variance parameter 2 2 2 The cumulative test statistic Z curve As mentioned in section 2 1 2 meta analysis test for statistical significance uses a Wald type test statistic This statistic is given by the log of the pooled intervention effect divided by its standard error and is commonly referred to as the Z statistic or the Z value Under the assumption that the two investigated interventions do not differ the null hypothesis the Z value will approximately follow a standard normal distribution a normal distribution with mean O and standard deviation 1 The larger the absolute value of an observed Z value the stronger is the statistical evidence that the two investigated interventions do differ If the absolute observed Z value is substantially larger than O it is usual to conclude that the observed difference between the effect of the two interventions cannot solely be explained by the play of chance In this situation the difference between the
34. true effect u Letting 7 denote the between trial variance the random effects model is defined as follows Yi ute s N 0 o u u E E NO 1 Where lt is the residual Sampling error for trial i and E is the difference between the true overall effect and the true underlying trial effect Collapsing the hierarchical structure in the above equations Y can be assumed to satisfy the distributional relationship Y N u of 7 Again the trial weights are defined as the reciprocal of the variance and so the trial weights in a random effects model become w of 7 The meta A analysed intervention effect u is obtained as a weighted average of the observed intervention effects of the individual trials oe User Manual for TSA Copenhagen Trial Unit 2011 and has variance Statistical significance testing is performed with the Wald type test statistic which is equal to the meta analysed intervention effect log scale for relative risks and odds ratios divided by its standard error A u VVar This test statistic is typically referred to as the Z statistic or the Z value Under the assumption that the two investigated interventions do not differ the Z value will approximately follow a standard normal distribution a normal distribution with mean O and standard deviation 1 This assumption is also referred to as the null hypothesis and is denoted Ho The corresponding two s
35. 1 1 4 Testing for futility before the information size has been reached It is also possible to use the TSA software to assess when an intervention is unlikely to have some anticipated effect Or in a clinical context to assess when an intervention has an effect that is smaller than what would be considered minimally important to patients Meta analyses are often used to guide future research Before embarking on future trials investigators need to know an accurate summary of the current knowledge If a meta analysis has found that a given intervention has no important effect investigators need to know whether this finding is due to lack of power or whether the intervention is likely to have no effect Using conventional thinking a finding of no effect is considered to be due to lack of power until an appropriate information size has been reached In some situations however we may be able to conclude earlier that a treatment effect is unlikely to be as large as anticipated and thus prevent trial investigators from spending resources on unnecessary further trials Of course the size of the anticipated intervention effect can be reconsidered and further research may be designed to investigate a smaller effect size Test A Test B statistic Statistic Figure 5 Examples of futility boundaries where the experimental intervention is not superior to the control intervention and too many trials may have been conducted A and whe
36. 12 2 1 1 Effect measures for dichotomous and continuous data ccc ceeecceeseeceeeteeeees 12 2 1 2 General fixed effect model and random effects model setup c cc eeeeeeeees 14 2 1 3 Approaches to random effects model meta analysis 0 0 0 0 ccc cccessecccesseeeeesseeeeens 16 2 1 4 Methods for handling Zero event trialS cece ccccessscceeesssececessseceeeenseeeeessseeeeeesaes 20 2 2 Adjusted significance testing and futility testing in cumulative meta analysis 22 2 2 1 The information size required for a conclusive meta analysis ccceeeseeeeeneees 24 2 2 2 The cumulative test statistic Z CUIVE 0 0 ccc ccccssecceeesseecceesseeceeessseeeeeenseeceeesseeeeeesaes 34 2 2 3 Problems with significance testing in Meta ANnalySIS cccccccescceeseeeeesteeeestseeees 35 2 2 4 The a spending function and trial sequential monitoring boundaries 37 2 2 5 Adjusted confidence intervals following trial sequential analySis cccecee 44 2 2 6 The law of the iterated logarithm o sesoosessessssssssesserressssressssressssreesssereessrressssressssreesss 47 2 2 7 The B spending function and futility boundaries sseseseeseseeseeesssrssseresssrrrrssrrrrssen 49 3 Installation and starting the TSA program iinei a a AA aa a aaa 55 Pal Od eregu NES ai a RU EE a EE ESS re 55 92x MSAA a es ee re ee ne ee ee 55 GG ee 11 T o 2 ene a nt NR ORC
37. Chapman amp Hall CRC Press 2000 Berkey C Mosteller F Lau J Uncertainty of the time of first sifnificance in random effects cumulative meta analysis Controlled Clinical Trials 1996 17 357 371 DerSimonian L Laird N Meta analysis in clinical trials Controlled Clinical Trials 1986 7 177 188 Sidik K Jonkman J Simple heterogeneity variance estimation for meta analysis Journal of Royal Statistical Society C 2005 54 367 384 Sidik K Jonkman J A comparison of heterogeneity variance estimators in combining results of studies Statistics in Medicine 2007 26 1964 1981 Brockwell S Gordon I A comparison of statitical methods for meta analysis Statistics in Medicine 2001 20 825 840 112 User Manual for TSA Copenhagen Trial Unit 2011 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 Brockwell S Gordon IR A simple method for inference on an overall effect in meta analysis Stat Med 2007 26 4531 4543 Sanchez Meca J Martin Martinez F Confidence intervals for the overall effect size in random effects meta analysis Psychological Methods 2008 13 31 48 Sidik K Jonkman J Robust variance estimation for random effects meta analysis Comp Stat Data An 2006 50 3681 3701 Viechtbauer W Bias and efficiency of meta analytic variance estimators in the random effects model J Educa Behav Stat 2005 30 261 293 Makambi KH The effe
38. Esmat 2008 lt I __intervention 17100 Controli 1 100 Rahman 1 2007 I _intervention 29 72 Control 21 72 ALFaleh 2004 I intervention 6 48 Control 448 Mangia 2005 I intervention 14 121 Control 14 120 Derbala 2006 I intervention 25 40 Control 9 40 Scoto 2008 _ intervention 3 26 Control 1052 Close PRR Ree Figure 18 Trials overview with bias risk check boxes If you click on the Trials overview button a new window with a list of all the trials in the csv file will open Each trial has an associated checkbox indicating whether the trial is designated as a low bias risk trial or not default is high bias risk You can change the designated bias risk of a trial by checking or un checking its bias checkbox Click on the Close button once you are done defining bias risks 63 User Manual for TSA Copenhagen Trial Unit 2011 Review TSA outcomes E x General meta analysis setting Pegylated interferon plus ribavirin Versus non pegylated interferon plus ribavirin Outcome 1 Outcome 2 OUECOme name adverse events Subgroup name adverse events leading to dose reduction Apply changes Cancel Create T54 filets Figure 19 Reviewing TSA outcomes in the RM5Converter Once you have checked all the comparisons outcomes and subgroups you want to include for your trial sequential analysis click on
39. Manual for TSA Copenhagen Trial Unit 2011 6 2 Random effects approaches 6 2 1 Formulas for the Biggerstaff Tweedie method Let fp t denote the probability density function of the DL estimate of 7 and let Fp t denoted the corresponding cumulative distribution function Fp t Defining the trial weights as a function of t by w t of t and using the obtained distribution of the estimate of tp the so called frequentist Bayes estimates of the trial weights can be obtained w t Elw t5 F tlo H w 0 f wO Oat subsequently yielding summary estimates of the overall population intervention effect 2 WY Her x 2w with variance l s Var Uz ea a RO i m G s Pi thereby ensuring that the variance of the summary effect estimate is adjusted with regard to the uncertainty associated with estimating the between trial variance 6 3 Trial sequential analysis 6 3 1 Exaggerated type error due to repeated significance testing By the laws of basic probability theory when data is tested twice over time and when an a of 5 is used as a threshold for both tests or a Z value of 1 96 the probability that the two interventions will be declared statistically significant under the null hypothesis is 103 User Manual for TSA Copenhagen Trial Unit 2011 Pr rejected Pr Z gt 1 96 or Z gt 1 96 Pr Z 1 96 Pr Z gt 1 96 Z lt 1 96 2 1 Pr Z lt 1 96 1 Pr Z lt 1
40. Radio buttons for choosing the information scale on which the cumulative significance testing is displayed lf one or two of the three scales Sample size event size or statistical information have not been selected in any of the added a spending boundaries they will automatically be greyed out in the nformation Axis area 4 5 Graphical options for TSA The Graph option in the TSA program allows you to display the Z curve and your constructed significance tests in relation to the strength of evidence i e accumulated number of patients events or statistical information It also provides a number of graphical editing options that may be useful when you _ 22 User Manual for TSA Copenhagen Trial Unit 2011 are preparing graphs for your article manuscripts To go to the Graph option click on the Graph tab to the right of the TSA tab as shown in figure 47 Figure 47 Click on the Graphs tab to view the Z curve and your constructed significance tests displayed in relation to the strength of evidence In the left side of the TSA program window you will find the Tests and boundaries Layout area the Set Graph Layout and two print options Print current graph and Generate TSA Report To the right of these areas you will find the graph displaying your Z curve and constructed significance tests In the Tests and boundaries layout area you will find a number of graphical editing options that allow you to chan
41. S OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM DAMAGES OR OTHER LIABILITY WHETHER IN AN ACTION OF CONTRACT TORT OR OTHERWISE ARISING FROM OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE UNDER NO CIRCUMSTANCES AND UNDER NO LEGAL THEORY WHETHER IN TORT CONTRACT OR OTHERWISE SHALL COPENHAGEN TRIAL UNIT BE LIABLE TO YOU OR TO ANY OTHER PERSON FOR LOSS OF PROFITS LOSS OF GOODWILL OR ANY INDIRECT SPECIAL INCIDENTAL OR CONSEQUENTIAL DAMAGES OR DAMAGES FOR GROSS NEGLIGENCE OF ANY CHARACTER INCLUDING WITHOUT LIMITATION DAMAGES FOR LOSS OF GOODWILL WORK STOPPAGE COMPUTER FAILURE OR MALFUNCTION OR FOR ANY OTHER DAMAGE OR LOSS The Trial Sequential Analysis software hereafter TSA to which this manual refers is in Beta Release Copenhagen Trial Unit has tested the TSA software extensively but errors may still occur Feedback is an important part of the process of correcting errors and implementing other changes so we encourage you to tell us about your experiences with this software To do so please send your feedback to tsa ctu dk User Manual for TSA Copenhagen Trial Unit 2011 Team member roles and contributions TSA was developed at The Copenhagen Trial Unit Copenhagen Denmark The team consisted of Kristian Thorlund KT Janus Engstrom JE Jorn Wetterslev JW Jesper Brok JB Georgina Imberger GI and Christian Gluud CG The roles and contributions of each team member are outl
42. The meta OS User Manual for TSA Copenhagen Trial Unit 2011 analytical data are considered as analogous to accumulating data in a single trial and the required statistical information is given by ISstatistical Z1 0 2 Z1 p 5 Where Ssztatisticas IS the actual attained statistical information in the meta analysis a is the desired maximum risk of type error Z1 a 2 is the standard normal 7 a 2 percentile 6 is the desired maximum risk of type II error Z1 g is the standard normal 1 6 percentile and 6 is some pre specified minimally relevant intervention effect The heterogeneity adjustment factor Trials included in a meta analysis often include patients from a wide span of population groups use different regimens of an intervention use different study designs and vary in methodological quality i e risk of bias or systematic error For all of these reasons it is natural to expect an additional degree of variation in meta analysis data compared to data from a single trial Such additional variation is referred to as heterogeneity or between trial variation Because increased variation can decrease the precision of results information size considerations must incorporate all sources of variation in a meta analysis including heterogeneity 4 111 One approach for incorporating heterogeneity in information size considerations is to multiply the required information size in a meta
43. User manual for Trial Sequential Analysis TSA Kristian Thorlund Janus Engstrom Jorn Wetterslev Jesper Brok Georgina Imberger and Christian Gluud Copenhagen Trial Unit Centre for Clinical Intervention Research Department 3344 Rigshospitalet DK 2100 Copenhagen Denmark Tel 45 3545 7171 Fax 45 3545 7101 E mail tsa ctu dk Contents B Rye 10 00s oem ener ene nmr eee nen mer ee een OR ee E eee ene eee i Team member roles and Contributions ccccccccsesseecccccccccscsueesseeecccecsecesueessesecceseseeeuuuseneeeeeeseeeeauaneees 2 DPT aroia a A A E A E AATE E E NA een E 3 1 Concepts and rationale behind trial sequential analysiS c cccccesssssssssccceceeceeeeeeeeeeeeeeeeeeeeeeeeeeeees 4 1 1 Random error in meta analysis cece cc cceeesssssecceeeeesseeeecceeessseeceeeeeesseeeeeceeesseeeeeeeeetseeeeees 4 1 2 Defining strength of evidence information SIZE eee cceccccceseceeesseceesseeeesseeceesseeeeseeeees 6 1 3 Testing for statistical significance before the information size has been reached 7 1 4 Testing for futility before the information size has been reached ccccccesceeesseeeeeees 9 kouma ae Viacctsvcreus ego eeu orate anes tected ae onsa ead oi ah bene aac a anes iattes 10 De WICLMOGOIO SO yOe MING NS Ais aoe sisisedctiniatsbaseccnmane a r ETE 12 2 1 Methods for pooling results from clinical trials 2 0 00 ccc ecccccessscceessseceesseeeeseecessseeeeeseees
44. Wedge Apply Inner Wedge E Bowen ew du B pending Function O Brien Fleming Required Information Size Information Size C User Defined f Estimate Type Leon BLS AL Power Relative Risk Reduction User Defined Low Bias Based if o Ingdente in Intenention arm E User Betined IngiGence in Controliarm Ue Figure 34 Alpha spending boundaries setting pop up window for dichotomous data meta analysis that appears when clicking on the alpha spending button For continuous data meta analysis the al pha spending boundaries setting window that appear when you click on the alpha spending button should similar to the one shown in figure 35 73 User Manual for TSA Copenhagen Trial Unit 2011 Add Continuous Alpha spending Boundary i j x Boundary Identifier tt AA Name hagas Testing Boundary Type C One sided Upper One sidedLower Two sided Type 1 Error D a spending Function O Brien Fleming Information Axis Sample Size Event Size Statistical Information Inner Wedge Apply Inner Wedge Required Information Size Information Size C User Defined i Estimate E User Befned Empirical Low Bias User Defined Empicel Low Bias Pa Pee LT Ey aT P eee Le ps a ape J Pi ci Wura AA r erea Heterogeneity Correction eof User Defined Model Variance Based Figure 35 Alpha spending boundar
45. a and ms the standard deviations of the two siaa User Manual for TSA Copenhagen Trial Unit 2011 intervention group mean responses sd4 and sdg and the total number of participants in the two intervention groups na and ng When the mean response is measured on the same scale for all trials comparative effectiveness is measured with the mean difference MD which is given by Ma mg The standard error of the mean difference is given by 2 2 SE MD sd _ sdz n n B When the mean response is not measured on the same scale mean responses can be standardised to the same scale allowing for pooling across trials The conventional approach is to divide the mean response in each trial by its estimated standard deviation thus providing an estimate of effect measured in standard deviation units Mean differences divided by their standard deviation are referred to as standardised mean differences SMD The TSA program does not facilitate meta analysis of SMDs Adjusted significance testing for SMD meta analysis would require information size calculation be calculated on the basis of expected mean differences reported in standard deviation units This effect measure does not resonate well with most clinicians and is therefore prone to produce unrealistic information size requirements 2 1 2 General fixed effect model and random effects model setup Assume we have k independent trials Let Yi be the observed intervention effect in
46. a priori estimate of the difference between means in the two intervention groups and o denotes the associated variance Alternatives to accumulating number of patients In meta analysis of binary data the information and precision in a meta analysis predominantly depends on the number of events or outcomes One can therefore argue that in the context of meta analysis information size considerations the required number of events is a more appropriate measure than the required number of patients Under the assumption that an equal number of patients are randomised to the two investigated interventions in all trials the required number of events may be determined as follows IS Events Pc IS 2 T Pe IS 2 where Sevents iS the required number of events for a conclusive and reliable meta analysis and Pc and Pe are as defined in the previous paragraph The statistical information Fischer information is a statistical measure of the information contained in a data set given some assumed statistical model In standard meta analysis comparing two interventions the statistical information is simply the reciprocal of the pooled variance In a meta analysis the statistical information is a theoretically advantageous measure because it combines three factors in one single measure number of patients number of events and number of trials This measure provides a simple approach to information size considerations in a meta analysis
47. ading to treatment discontinuation 14 ce G lt comparison 2 Subgroup analysis Convert to TSA file s Trials overview Figure 17 Check the box tree structure in RM5Converter After you have exported your data to a csv file open RM5Converter by double clicking on the icon Go to the menu bar and select File gt Open A check box tree structure will appear in the application window figure 17 The data is structured the same way as in Review Manager v 5 For each comparison there can be multiple outcomes and each outcome represents a meta analysis If a meta analysis contains subgroup analyses the subgroups will be listed under each outcome If a comparison is checked all outcomes under that comparison will automatically be checked Also if an outcome is checked all subgroups under that outcome will automatically be checked All trials under each comparison outcome or subgroup are automatically included You dont have to convert everything listed under a given comparison You can uncheck the comparisons outcomes and or subgroups that you do not want to convert If the trials in the subgroups are unique you will be presented with the option of combining these subgroups into a single analysis 260 User Manual for TSA Copenhagen Trial Unit 2011 x Trial List Identifier _Low Bias Risk Data Roffi zoos __intervention 17 24 Contrai 5 6 Horsmans 2008 L intervention 10 23 Control 6 26
48. al u The overall true intervention effect H The pooled intervention effect o The variance of of The variance of Y T The between trial variance toi The DerSimonian Laird estimate for the between trial variance Tsj The Sidik Jonkman estimate for the between trial variance The pooled odds ratio of excluding zero event trials D The cumulative standard normal probability distribution function 110 User Manual for TSA Copenhagen Trial Unit 2011 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Reference List Pogue J Yusuf S Cumulating evidence from randomized trials utilizing sequential monitoring boundaries for cumulative meta analysis Controlled Clinical Trials 1997 18 580 593 Pogue J Yusuf S Overcoming the limitations of current meta analysis of randomised controlled trials Lancet 1998 351 47 52 Sterne JA Davey SG Sifting the evidence what s wrong with significance tests British Medical Journal 2001 322 226 231 Thorlund K Devereaux PJ Wetterslev J et al Can trial sequential monitoring boundaries reduce spurious inferences from meta analyses International Journal of Epidemiology 2009 38 276 286 Trikalinos TA Churchill R Ferri M et al Effect sizes in cumulative meta analyses of mental health randomized trials evolved over time Journal of Clinical Epidemiology 2004 57 1124 1130 Wettersl
49. al meta analysis models the fixed effect model and some variants of the random effects model and methods for handling zero event data In section 2 2 we describe the methods for adjusting significance when there is an increased risk of random error due to weak evidence and repeated significance testing We do not describe the more advanced part of this methodology in detail Rather this chapter is intended to provide users with an intermediate level conceptual understanding of the issues addressed in chapter 1 2 1 Methods for pooling results from clinical trials 2 1 1 Effect measures for dichotomous and continuous data The TSA program facilitates meta analysis of dichotomous binary data and of continuous data Dichotomous data are data that is defined by one of two categories e g death or survival Continuous data are data that is measured on a numerical scale e g blood pressure or quality of life scores For each type of data there are various measures available for comparing the effectiveness of an intervention of interest 1 Dichotomous data effect measures Assume we have k independent trials comparing two interventions intervention A vs intervention B with a dichotomous outcome Such trials will typically report the number of observed events e g deaths in the two intervention groups ea and eg and the total number of participants na and np in the two intervention groups For dichotomous data the interve
50. alse negative result This phenomenon is commonly known as multiplicity due to repeated significance testing In meta analysis it is important to minimize the risk of making a falsely positive or falsely negative conclusion Pooled intervention effects in meta analysis are typically assessed on the basis of P values Meta analysts must decide on the threshold at which a P value is sufficiently small to justify a positive conclusion Below this threshold a conclusion is considered statistically significant At a given time any threshold involves a trade off between the risk of observing a false positive result type error and the risk of observing a false negative result type II error For example if the threshold for statistical significance in figure 2 horizontal dashed line had been moved up the 5 User Manual for TSA Copenhagen Trial Unit 2011 chance of observing a false positive result figure 2 A would have diminished while the risk of observing a false negative result figure 2 B would have increased When conventional significance tests are performed at early stages and or at multiple times these maximum risks are distorted as illustrated in figure 2 Thus any inferences about statistical significance should be made in relation to the strength of the evidence The strength of evidence should be measured using the accrued number of patients observed number of events in the included tr
51. analysis by some heterogeneity adjustment factor Recently a similar heterogeneity adjustment factor has been proposed for estimating the sample size in a single clinical trial The heterogeneity adjustment factor is conceptualised through the underlying assumptions that we make for our meta analysis model In the fixed effect model it is assumed that all included trials can be viewed as replicates of the same trial with respect to design and conduct Thus the required information size for a fixed effect meta analysis to be conclusive may effectively be calculated in the same way as the required sample size for a single clinical trial In the random effects model we assume that the included trials come from a distribution of possible trials with respect to design and conduct By BE User Manual for TSA Copenhagen Trial Unit 2011 definition the variance in a random effects model is always greater than that in a fixed effect model A heterogeneity adjustment factor must therefore account for the increase in variation that a meta analysis incurs from going from the fixed effect assumption to the random effects assumption An accurate adjustment can be achieved by making the heterogeneity adjustment factor equal to the ratio of the total variance in a random effects model meta analysis and the total variance in a fixed effect model meta analysis The heterogeneity adjustment factor is therefore always equal to or greater than
52. and non inferiority futility boundaries In this example the required information size is 4000 At approximately 3000 patients the Z value falls within the inner 253 User Manual for TSA Copenhagen Trial Unit 2011 wedge and a conclusion can be made the intervention effect is not greater than the one anticipated a E bs O Brien Fleming boundaries 1 96 P 6 03 Sumber of patients Z 1 96 Figure 12 Example of a meta analysis with repeated non superiority non inferiority and significance testing boundaries 54 User Manual for TSA Copenhagen Trial Unit 2011 3 Installation and starting the TSA program 3 1 Prerequisites The Trial Sequential Analysis TSA software is a Java program and will therefore run on any operating system that supports Java Microsoft Windows Mac OS UNIX Linux etc The TSA software requires that you have the latest or at least a recent version of the Java Runtime Environment JRE installed on your computer You can download the JRE for free at Wwww java com At the time of writing August 2011 the latest JRE version is 1 6 The TSA software runs well with this version 3 2 Installation The TSA software is delivered in a ZIP archive Use any archive tool such as WinRAR or GZIP to unpack the archive In the archive you will find three files named TSA jar RM5Converter jar and TEMPLATES TPL along with two folders named ib and samples TSA
53. andom error and imprecision only cause problems if statistical tests and intervention effect estimation are employed at stages where the magnitude of the random error or imprecision is extreme enough to yield spurious ae User Manual for TSA Copenhagen Trial Unit 2011 Statistical inferences In figure 2 A significance testing at times X and X3 would result in a false declaration of statistical significance i e a false positive result whereas significance testing at X2 and X4 would not Thus only at times X and X3 is the impact of random error extreme enough to yield spurious statistically significant results In figure 2 B significance testing at X1 and X2 could have resulted in a false declaration that the interventions under investigation were not significantly different i e a false negative result whereas significance testing at X3 and X4 would not Thus only at times X and X2 is the imprecision of a magnitude that causes spurious absence of statistical significance Test A Test B statistic statistic Significant Significant el ae tne ee ene eee ee gt We e e e e e o I al Non significant Figure 2 Examples of false positive and false negative statistical test results over time in two randomised clinical trials A and B The more statistical tests that are employed throughout the accumulation of additional data the higher the likelihood of observing a false positive or f
54. apy is very low Sometimes there are zero outcome events recorded in a group In this situation ratio effect measures RR and OR will not give meaningful estimates of the intervention effect One solution for this problem is to add some constant s to the number of events and non events in both intervention groups This approach is known as continuity correction Several approaches to continuity correction have been proposed in the meta analytic literature Constant continuity correction The constant continuity correction is a simple method and is the most commonly used in the meta analytic literature The method involves adding a continuity correction factor a constant to the number of events and non events in each intervention group Group Events No Events Total Intervention 0 20 20 Control 5 20 25 Table 1 Example of a zero event trial Consider the zero event trial example in table 1 If for example the constant continuity correction method uses a correction factor of 0 5 the number of events in the intervention group becomes 0 0 5 0 5 the number of non events in the intervention group becomes 20 0 5 20 5 the number of events in the control group becomes 5 0 5 5 5 and the number of non events in the control group becomes 20 0 5 20 5 Because the total number of patients is the number of events plus the number of non events the total number of patients after constant continuity correction with the constant 0 5 b
55. aries it is possible to approximate how many patients should be randomised in the next trial to make the meta analysis cross either of the two boundaries A recent methodology paper illustrated this approach using a meta analysis of isoniazid chemoprophylaxis for preventing tuberculosis in HIV positive patients This meta analysis included nine trials 2911 patients and 131 events and yielded a pooled relative risk of 0 74 95 Cl 0 53 to 1 04 The estimated inconsistency and diversity were both 0 Risk ratio Risk ratio Study IV Random 95 Cl Weight IV Random 95 Cl Pape 1993 0 70 0 15 3 28 4 9 Wadhawan 1997 0 44 0 22 0 91 22 8 a Whalen 1997 0 74 0 30 1 80 14 7 Gordin 1997 0 49 0 12 1 95 6 1 Hawken 1997 0 72 0 29 1 78 14 1 Mwinga 1997 0 81 0 30 2 19 11 8 Fitzgerald 2001 1 32 0 38 4 56 7 5 Rivero 2003 0 70 0 16 3 01 5 4 Mohammed 2007 1 56 0 60 4 06 12 7 Total 95 Cl 0 74 0 53 1 04 100 0 ee Total events j Z f 0 sige alain ag 0102 05 1 2 5 10 Test for overall effect Z 1 74 P 0 08 Faora Hz Eavcra coniiole Figure 62 Forest plot of the individual trial effects of isoniazid chemoprophylaxis vs control for preventing tuberculosis in purified protein derivative negative HIV infected individuals We estimated the required information size for detecting a 25 relative risk reduction in tuberculosis with an alpha 5 and beta 20 80 power The required information size was b
56. ased on the assumption of a 5 control group incidence rate approximately the median rate across trials We also heterogeneity corrected the required information size assuming 20 diversity D This yielded a required information size of 10 508 patients Statistical 97 User Manual for TSA Copenhagen Trial Unit 2011 monitoring boundaries and futility boundaries were subsequently constructed according to the set error levels and the required information size Cumulative Z ncore Moclerate evidence 10508 E Ds sae F HE F as zg f ei Number of Sa aa patients p7 TS _ Linear scaled aa 3 z aii E cH 4 BE oii eH 5 7 m DO r Fa A 7 ae lt i o z Figure 63 Prospective trial sequential analysis of isoniazid vs control for preventing tuberculosis To the left the red inward sloping dashed lines make up the trial sequential monitoring boundaries To the right the outward sloping red dashed lines make up the futility region The solid blue line is the cumulative Z curve The last line on the cumulative Z curve represents an imagined trial that makes the meta analysis conclude that the isoniazid prevents tuberculosis To estimate how many patients would needed to be randomised in a future clinical trial to make the meta analysis conclusive we approximated the number of patients in an imaginary future trial that would make the cumulative Z curve cross the moni
57. atal 25 Cl 443 sag a 21 45 Total events 69 Otf pump 125 On pumpi Test for heterogeneity Cn 13 36 df 3 P 0 00 F 77 5 Test for owerall elled Z 2 05 P 0 03 Of Trials with adequale ramdomisaiion lack binding Penta 2001 Lf11 3 11 GE 2001 zamar 2002 a ao 10430 2 33 2002 Raja 2003 12 1450 55 150 B03 z003 Awan 2004 HAJS 5 32 2 17 Z004 Gerda 2004 aF eo 7490 1 68 20 Legare 2004 48 150 ag 150 7 53 2004 Lingaas 2004 18 60 20 60 5 63 2004 PARAGUE 4 2004 41 208 147132 7 37 z004 Kobayashi 2005 19 81 i3 86 5 30 Z005 Michaux 2006 gy 25 ay 25 3 63 2006 Niranjan 2006 3 40 12 40 4 79 2006 Papareila 2006 4 15 zy 16 1 20 2006 Jares 2007 2 10 2 10 0 95 2007 Ozkera 2007 2 22 ay 22 1 02 2007 Suntotal 95 Cl 317 307 i TAE Tolal events 166 Of pumpi 241 iOnmpumgi Test for heterogeneity Chi 25 66 df 13 F 0 02 F 49 9 Test for Overall efie Z 2 25 F 0 02 03 Trials with unces randomisation andor lack of Blinding Czermy 2000 2 18 3 15 1 08 z000 Kochamba 2000 ee uy 23 2 52 2000 Czarny 2001 5 ao 7 40 2 27 ZOOL Guler 2001 bag av 1g z001 Lee 2003 Tao 127 30 4 55 2003 Munareilo 2003 19 88 31 48 6 04 2002 Velssaris 2003 6 27 af 27 2 49 2003 Synneren 2004 1 26 12 26 4 71 z004 Ginema A006 o ao J712 2006 Manscalco 2006 10 45 12 35 4 17 2006 Quaniers 2006 a 40 137 40 3 30 2006 Subtotal 25 Cl ano 350 gt 23 67 Total events 70 Ctf pump 104 Cnpump Test for haberogenety Chi 1 41 a 6 P 0 89 F
58. atistical information left graph in figure 65 The O Brien Fleming type efficient score sequential boundaries were recently explored empirically and through simulation A study by van der Tweel and Bollen compared O Brien Fleming significance boundaries the ones implemented in the TSA software to the O Brien Fleming type efficient score sequential boundaries in six meta analysis These six meta analyses were the ones initially and randomly selected as illustrative examples in the methods paper proposing the information size heterogeneity correction for trial sequential analysis which is described in section 2 2 1 of this manual 105 User Manual for TSA Copenhagen Trial Unit 2011 Tweel and Bollen found that the two methods were identical in testing for significance A simulation study by Higgins et al investigated the type error and adjusted confidence interval coverage associated with the O Brien Fleming type efficient score sequential boundaries under a number of random effects model approaches They found that under this design the conventional DerSimonian Laird random effects model and the Biggerstaff Tweedie approach did not yield satisfactory results but a semi Bayesian approach utilizing an informative Gamma distribution on the between trial variance did Another example of the efficient score sequential boundaries is the triangular test proposed by Whitehead The boundaries produced from this method a
59. bove the graph To see the graphical representation of the calculated law of the iterated logarithm penalties select the Penalised Tests tab above the graph figure 53 Adjusted Boundanl s Penalized Tests Figure 53 View boundaries test or penalised test graph 4 6 Exploring diversity across trials The TSA program also provides an option for exploring diversity estimates and comparing weights across the three random effects models DL SJ and BT These options are available in the Diversity tab figure 54 86 User Manual for TSA Copenhagen Trial Unit 2011 Meta analysis Trials T54 crag Diversty gt Figure 54 Click on the Diversity tab to explore diversity estimates and compare weights across random effects models After you click on the diversity tab a screen similar to the one shown in figure 55 should appear In the upper part of the screen the weights and weight percentages for each trial rows using each of the available models columns are displayed in the lower left corner The following things are displayed for each of the three random effects models the estimate of inconsistency and its corresponding heterogeneity correction 1 1 I the estimate of diversity D and its corresponding heterogeneity correction 1 1 D and the estimate of between trial variance 1 The estimate of inconsistency is only displayed for the DL model Note that the estimate of between trial variance is the same f
60. ce tests applying TSA ccccccccscccesssecceesssseceeessseeeeenes 70 4 4 1 Adding a significance TESTE ec ccseccccessceeccessseeccessseeecesssseeceeesseeceeesseeeesessseeeeeesaes 71 4 4 2 Editing and deleting a significance test eee cccessseccceesseeeceessseeceeesseeeeeesseeeeens 78 4 4 3 Adding and loading significance test templates 0 0 0 0 ccesccccesseeccessseeecestseeeens 79 4 4 4 Performing the significance test calculations c ce ccccscccessseceeeessecceeesseeeeeesteeeeeens 80 4 5 Graphical ptions for TSA iciscd dsuie nbenicesadeiinetatvauke Seioancondandaederdaagettiaatoveadaalonaradeaetadovacesdatands 82 4 6 Exploring diversity across trials 2 0 0 cece ccccccesssssccceeeessceeeccceessseeeceeeesseeececesaseeeeeeeeessaeeeees 86 5 TSA example AD Pl CATIONS 5 sc s ace dances E dane nie casa ana sowie ateaotsae ee aw eves ans 88 STA Wad B 2 E oe ee ne ee ees ae SA A Oe eR A A E ae ere ee eee EA AATE AAE AAT 88 5 2 Avoiding false POSITIVES 00 0 cece cccessscccesssscccessssceccesssseccessssseceesssseceeeeseeecesesseeceeseseeeeeesaes 88 5 3 Confirming a positive result oo cee cccesssecceessseecceseseecceessseeceeseseeceessseeecessseeeceessseeeesenaes 90 5 3 2 Avoiding early overestimates 0000 0 cece ccccsssccccceeesssseecceeeeeseeeeecceenssseeeceeeeesseeeceeensaaaeeees 93 gak TESNO FOF TUY ct acces shen st set seas so tncatadentasdbaoitoadonacata lee A 95 5 5 Estimating the sample size of a new cli
61. cient scores In sequential analysis of efficient scores information is measured as Statistical information i e Fischer information The efficient score is plotted y axis against the statistical information x axis and monitored with some boundaries Just as with the alpha spending and beta 104 User Manual for TSA Copenhagen Trial Unit 2011 spending based boundaries the sequential method for monitoring efficient scores produce superiority inferiority and futility boundaries Examples of such boundaries are illustrated in figure 65 below Figure 65 Illustration of two types of monitoring boundaries from sequential meta analysis of efficient scores The left graph illustrates what would correspond to an O Brien Fleming alpha spending significance boundaries and O Brien Fleming beta spending futility boundaries The right graph illustrates what corresponds to what is typically Knows and Whitehead s triangular boundaries The latter is designed to minimize total risk of statistical error i e type and type Il error together Just like different a spending functions yield different types of adjusted significance boundaries the triangular test can be used to construct different types of boundaries and similarly for beta spending functions and futility boundaries For example a special case of the triangular test yields boundaries that are equivalent to the O Brien Fleming boundaries when accumulating st
62. clusions made using TSA show the potential to be more reliable than those using traditional meta analysis techniques Empirical evidence suggests that the information size considerations and adjusted significance thresholds may eliminate early false positive findings due to imprecision and repeated significance testing in meta analyses 4 1112 Alternatively one can penalise the test statistic according to the strength of evidence and the number of performed significance tests the law of the iterated logarithm Simulation studies have demonstrated that penalizing test statistics may allow for good control of the type error in meta analyses The following manual provides a guide both theoretical and practical for the use of Copenhagen Trial Units TSA software Chapter 2 provides a technical intermediate level overview of all the methodologies incorporated in the TSA software Chapters 3 5 are practical chapters on how to install use and apply the TSA software User Manual for TSA Copenhagen Trial Unit 2011 2 Methodology behind TSA TSA combines conventional meta analysis methodology with meta analytic sample size considerations i e required information size and methods already developed for repeated significance testing on accumulating data in analysis methodology used to pool data from a number of trials The description in section 2 1 covers effect measures for dichotomous and continuous data statistic
63. ct Z 0 44 P 0 66 Total 95 CI 19485 1974 100 00 Total events 48 Off pump 48 On pumpi Teast for haterogeneity Chi 12 97 df 22 P 0 93 F OF Test for overall affect Z 0 17 P 0 86 0 001 0 01 0 1 1 i0 100 1000 Favours off pump Favours on pump Figure 60 Forest plot of the effect of off pump vs on pump CABG on myocardial infarction 5 4 Testing for futility The example of the off pump vs on pump CABG meta analysis can also be used to illustrate testing for futility this time using the outcome of myocardial 95 User Manual for TSA Copenhagen Trial Unit 2011 infarction MI Occurrence of MI was reported in 44 trials including 4303 patients No significant difference occurred between off pump vs on pump surgery RR 1 06 95 Cl 0 72 to 1 54 Figure 60 and this result was independent of risk of bias No statistical heterogeneity was detected 0 Nineteen trials 909 patients were zero event trials When zero event trials were continuity corrected there was also no noticeable change in the results RR 1 05 95 CI 0 74 to1 48 S Required Information Size 5942 Favors Off pump Number of patients Linear scaled Favors On pump Figure 61 The heterogeneity adjusted required information size to demonstrate or reject a 33 relative risk reduction a priori estimate of myocardial infarction MI with an occurrence of MI in the on pump group of 3 9 an alpha of 5 and a
64. ct of the heterogeneity variance estimator on some tests of efficacy J Biopharm Stat 2004 14 439 449 Biggerstaff B Tweedie R Incorporating variability in estimates of heterogeneity in the random effects model in meta analysis Statistics in Medicine 2009 16 753 768 Sweeting M Sutton A Lambert P What to add to nothing Use and avoidance of continuity corrections in meta analysis Statistics in Medicine 2004 23 1351 1375 Armitage P Sequential analysis in therapeutic trials Annual Review of Medicine 1969 20 425 430 Pocock S Group sequential methods in the design and analyss of clinical trials Biometrika 1977 191 199 Whitehead J The design and analysis of sequential clinical trials John Wiley amp Sons 2000 Whitehead A Whitehead J A general parametric approach to the meta analysis of randomized clinical trials Statistics in Medicine 1991 10 1665 1677 Higgins JPT Thompson S Quantifying heterogeneity in a meta analysis Statistics in Medicine 2002 21 1539 1558 Federov V Jones B The design of multicentre trials Statistical Methods in Medical Research 2005 14 205 248 Ioaniddis J Patsopoulos N Evangelou E Uncertainty in heterogeneity estimates in meta analysis British Medical Journal 2007 335 914 916 Jackson D The implications of publication bias for meta analysis other parameter Statistics in Medicine 2006 25 2911 2921 Chan AW Hrobjartsson A Haahr M Gotzsche P Altman D Empirical evidence for se
65. ction available in TSA is the O Brien Fleming function You will need to input the necessary components for the required information size calculation You will have the option to define the required information as any arbitrary number you may have obtained independent of the TSA software To submit your own value for IS check the radio button User defined and type in the required IS You also have the option to estimate the required IS according to the methods delineated in section 2 2 1 To use TSA to calculate the required IS check the radio button Estimate The required IS estimate will automatically be generated with respect to the type of information you are accumulating For example if you selected sample size under Information Axis the required information size will the generated as the required number of patients in the meta analysis The IS calculation will automatically be based on the maximum type error you defined for the a spending boundary but you will need to enter your desired maximum type II error minimum desired power 1 type Il error into the input field Power You have two options for adjusting the required information size for heterogeneity in the meta analysis The first option is to base the heterogeneity adjustment on the estimated ratio between the variance in the selected random effects model and the variance in the fixed effect model see section 2 2 1 To use this option c
66. ction method pull the intervention effect estimates towards the null effect i e towards O for risk differences and toward 1 for ratio measures An alternative continuity correction is the empirical continuity correction which pulls the intervention effect estimate towards the meta analysed effect For example let be the odds ratio of the meta analysis that does not include the zero event trials and let R be the randomisation ratio in the trial that needs continuity correction The continuity correction factor for the intervention group CF and the continuity correction for the control group CFc can be approximated with the following formulas LO User Manual for TSA Copenhagen Trial Unit 2011 Ci C R 0 CF AC R 0 under the restriction that the two continuity corrections add up to some constant C 4 2 2 Adjusted significance testing and futility testing in cumulative meta analysis Adjusted significance testing in cumulative meta analysis has two goals it must measure and account for the strength of the available evidence and it must control the risk of statistical errors type error and type II error when repeated significance testing on accumulating data occurs Quantifying the strength of the available evidence necessitates the definition of a goal post 49 11 1223 In the TSA programme TSA the strength of available evidence is measured and considered by calculating a required
67. d In this situation the first meta analysis yields a Z value Z4 and the meta analysis update yields another Z2 If the first meta analysis yields a Z value larger than 1 96 the two investigated interventions are declared significantly different However if the first meta analysis is not significant i e Z lt 1 96 the two interventions can still be declared statistically significant if the meta analysis update yields a Z value larger than 1 96 i e if 2221 96 By the laws 236 5 User Manual for TSA Copenhagen Trial Unit 2011 of basic probability theory the probability that the two interventions will be declared statistically significant under the null hypothesis is Pr H rejected Pr Z gt 1 96 or Z gt 1 96 Pr Z gt 1 96 Pr Z 1 96 Z lt 1 96 It can be shown that this expression is always larger than the desired 5 see appendix A 3 1 In general repeated significance testing using single test thresholds will always lead to an exaggeration of the type error and the larger the number of repeated significance tests employed on accumulating data the worse the exaggeration of the type error becomes For meta analysis data simulation studies have demonstrated that repeated significance testing result in a type error of 10 to 30 when the conventional a 5 threshold 1 96 is used to test for statistical significance at every update 8 10 31 2 2 4 The a spending function and trial seq
68. daries are applied to a sequence of trials and we therefore refer to them as trial sequential monitoring boundaries The combination of meta analysis and trial sequential monitoring boundaries is referred to as trial sequential analysis Trial sequential monitoring boundaries require pre specification of the k maximum type error risks a7 k as well as intensive numerical integration for their application One simple method for assigning values for the aj Ax type error risks is the a spending method or a spending function This method is implemented in the TSA program The a spending function is a monotonically increasing function of time that can be used for appropriately assigning maximum type error risks ay Ax at each 38 User Manual for TSA Copenhagen Trial Unit 2011 significance test according to the amount of information accumulated The independent variable is defined by the information fraction IF this is calculated by dividing the accumulated information by the required information size e g the accumulated number of patients divided by the required number of patients The dependent variable the function is the cumulative type 1 error this gives the amount of error that should be considered the maximum when defining significance at the given IF As IF increases i e as the amount of accumulated information increases the size of acceptable type 1 error also i
69. e and Pc may be entered into the formula for the required information size Drawing inference about anticipated realistic intervention effects from one intervention area to another may be problematic because an a priori estimate may often represent poor approximations of the truth The clinical trial literature abounds with examples of sample size calculations based on overly optimistic anticipated intervention effects There is no reason why this should be any different for meta analysis information size calculations lf randomised trials have already investigated the effect of an intervention then a collection of such estimates may be used to better quantify an anticipated intervention effect However not all trials provide valid estimates and caution should be taken to ensure the validity of intervention effects estimates utilised for estimating some anticipated intervention effect Many trials yield overestimates of investigated intervention effects due to selective outcome reporting bias and risks of bias i e systematic errors due to inadequate generation of the allocation sequence inadequate allocation concealment inadequate blinding loss to follow up or _ other mechanisms Such trials may be classified as trials with high risk of bias Conversely trials that are likely to yield valid intervention effect estimates may be classified as trials with low risk of bias If evidence on the effect of the investiga
70. e patient population for which the a priori minimally important difference apply When the required information size is to be defined by the required number of patients or events the problem of unpredictable heterogeneity may be dealt with by anticipating some appropriate maximum degree of heterogeneity and adjusting the required information size accordingly The apparent limitation of this approach is that the degree of expected heterogeneity is both difficult to guess and estimate when only a few clinical trials are available Although we recommend sensitivity analysis on the degree of heterogeneity adjustment such analyses may still be inappropriate if the anticipated degree s of heterogeneity does not reflect the actual degree of heterogeneity which the meta analyses will incur as more trials are accumulated When the required information size is defined by the required statistical information the formula for the required information size does not require an estimate of the anticipated degree of heterogeneity Rather the actual information in the meta analysis the estimated statistical information directly incorporates the heterogeneity through the estimated between trial variation an 34 User Manual for TSA Copenhagen Trial Unit 2011 This however presents a limitation in that the accumulated statistical information is only reliable to the extent the estimate of the between trial variance is reliable Possible solutions to
71. e used data from a review comparing smoking cessation rates in patients receiving hospital contact plus follow up for less than 1 month with patients receiving no intervention In the systematic review the interventions and length of follow up differed substantially across the included trials The authors therefore used the following categorisation of intervention intensity 1 Single contact in hospital lasting lt 15 minutes no follow up support 2 One or more contacts in hospital lasting gt 15 minutes no follow up support 3 Any hospital contact plus follow up lt 1 month 4 Any hospital contact plus follow up gt 1 month The meta analysis of intervention intensity 3 included six trials 4476 patients and 628 events The fixed effect model yielded a pooled relative risk of 1 05 95 CI 0 91 to 1 21 the meta analysis of odds ratios showed a similar result The estimated inconsistency I was 9 and the estimated diversity D was 10 We performed a retrospective trial sequential analysis by re doing a conventional meta analysis on the accumulating data each time a new trial was published The first published trial yielded a relative risk of 1 47 95 CI 1 05 to 2 05 After the second trial the pooled relative risk was 1 33 95 Cl 1 02 to 1 75 The meta analysis comparing intervention intensity category 3 see above with control was therefore nominally statistically significant after the first two trials 88
72. ecomes 20 5 0 5 21 in the intervention group and 20 5 5 5 26 in the control group sye User Manual for TSA Copenhagen Trial Unit 2011 If for example a correction factor of 0 1 is used the number of events and total number of patients after continuity correction would then be 0 1 and 20 2 in the intervention group and 5 1 and 25 2 in control group Review Manager Version 5 uses constant continuity correction with the constant 0 5 Simulation studies have demonstrated problems with the use of this constant it yields inaccurate estimates when the randomisation ratio is not 1 1 and it produces confidence intervals that are too narrow Reciprocal of opposite intervention group continuity correction Another potential continuity correction method is to add the reciprocal of the total number of patients in the opposite intervention group to the number of events and non events This type of continuity correction is also commonly referred to as treatment arm continuity correction In the example in table 1 the correction factor for the intervention group would be 1 25 0 04 and the correction factor for the control group would be 1 20 0 05 This continuity correction method yields 0 04 events and 20 04 patients in the intervention group and 5 05 events and 25 05 patients in the control group Empirical continuity correction Both the constant continuity correction method and the treatment arm continuity corre
73. ect File gt Save as If you wish to continue working on an already created TSA file go to the menu bar select File gt Open and locate the TSA file on which you wish to continue working 4 1 3 Importing meta analysis data from Review Manager v 5 To import meta analysis data saved in a Review Manager v 5 file rm5 presently however for dichotomous outcomes only you will need to use the separate software application RM5Converter which is included in the Zip archive that you downloaded before installing TSA see chapter 3 60 User Manual for TSA Copenhagen Trial Unit 2011 4 Export Analysis Data Wizard Export Anabhysis Data Wizard Which analyses would you like to export Analyses Data and analyses d 1 Pegylated interferon alpha 2a versus pegylated interferon alpha 2b 4 1 1 Sustained virological response Z 1 2 Liverrelated morbidity plus all cause mortality 4 1 3 Adverse events leading to treatment discontinuation f 2 Subgroup analysis t2 2 1 Sustained virological response according to genotype 4 2 2 Sustanied virological response according to treatment history Cancel Figure 16 Pop up export analysis data wizard window in Review Manager v 5 which allows you to select the meta analyses you wish to export presently however only for dichotomous outcomes as a csv file The RevMan file to be converted is from the Cochrane review Pegylated interferon alfa 2a versus pegylated interfer
74. ed meta analysis estimates of these trials as your anticipated relative risk reduction To use this option select the Low bias Based option To set the anticipated mean difference and variance for a continuous data meta analysis you only need to fill in two fields Mean difference and Variance If you have categorised some of your included trials as low bias risk trials you may use the pooled meta analysis estimates of these trials as your anticipated mean difference and variance again by selecting Low bias Based option You also have the option to use the pooled estimate of all included trials regardless of bias risk as your anticipated variance To use all trials select the Empirical option ae oe User Manual for TSA Copenhagen Trial Unit 2011 When you have named your a spending boundaries defined the hypothesis test settings and the parameters for your information size calculation press the Add button to add the boundaries After you have added your a spending boundaries you will need to define when the meta analysis was previously subjected to significance testing Go to the Interim looks to the right of the list of adjusted significance tests and check or uncheck the trials after which significance testing were previously performed In figure 36 trials 2 4 and 5 have been checked and trials 1 and 3 have been unchecked meaning that three meta analyses including significance testing were
75. ed testing is analogous to the desire to control the type error Multiple testing increases the actual amount of error and we need to find a technique to control this increase Just as it is caused by the same phenomenon the problem of an increased type Il error can be managed using a similar solution In section 5 User Manual for TSA Copenhagen Trial Unit 2011 2 2 3 the alpha spending function was described as a technique which can be used to create reasonable boundaries for significance testing Similarly the problem of finding repeated non superiority testing thresholds which will ensure good control of the type II error can be solved by introducing the B spending function The spending function is a monotonically increasing function of time which is used to appropriately assign maximum type II error risks 61 Bx at each non superiority test according to the amount of information accumulated The f6 spending function is a function of the information fraction IF the accumulated information divided by the required information size and it is only defined from O0 to 1 The 6 spending function of O is always equal to 0 6 0 0 and the spending function of 1 is always equal to 6 B 7 6 At any point between O and 1 the 6 spending function is equal to the total maximum type II error risk that has arisen from the thresholds chosen for all non superiority tests until and including the th test In other words the spending func
76. effect thus facilitating adjusted confidence intervals See appendix section 6 2 1 Which random effects approach may be best The SJ and BT approaches both offer relative merits over the DL approach However these methods have their own limitations and are unlikely to be superior in all cases The SJ estimator may overestimate the between trial 18 User Manual for TSA Copenhagen Trial Unit 2011 variance in meta analyses with mild heterogeneity thus producing artificially wide confidence intervals The BT approach has been shown to provide similar coverage as the confidence intervals from the DL approach in meta analyses with small unbiased trials However when the included trials differ in size and some small trials are biased the BT approach will put appropriately high weights on the larger trials while still accounting for heterogeneity This point is important because a common critique of the DL random effects model is that small trials are often assigned artificially large weights in heterogeneous meta analyses A commonly applied and unsatisfactory solution is to use the fixed effect model instead By doing so the pooled estimate may incur less bias from the inappropriate weighting scheme but the confidence intervals will also be artificially narrow because they do not account for heterogeneity The BT approach mitigates the bias incurred from inappropriate random effects model weighting while still accounting
77. emonstrate or reject a 20 relative risk reduction a priori estimate of atrial fibrillation with a control group proportion of 27 6 an alpha of 1 and a beta of 10 is 7150 patients vertical red dashed line The red dashed inward sloping line to the left represents the trial sequential monitoring boundaries which are truncated for the first 14 trials The solid blue line is the cumulative Z curve All information sizes were derived to ensure a maximum type error of 1 and a maximum type II error of 10 i e 90 power All information sizes were heterogeneity adjusted using the estimate of diversity D Both information sizes were derived assuming an event proportion of 27 6 in the on pump group median event proportion in this control group The cumulative Z curve crossed the monitoring boundaries constructed from both information size calculations Figure 58 and 59 thereby confirming that off pump CABG is superior to on pump CABG in reducing atrial fibrillation 92 User Manual for TSA Copenhagen Trial Unit 2011 Cumulate acore Required Information Size 1964 3374 Number of patents Figure 59 The heterogeneity adjusted required information size to demonstrate or reject a 36 9 relative risk reduction low bias risk trial estimate of atrial fibrillation with a control group proportion of 27 6 an alpha of 1 and a beta of 10 is 1964 patients vertical red dashed line The red dashed inward slop
78. equired information size Figure 11 Example of a meta analysis including repeated non superiority red line and significance brown line testing The cumulative Z curve for the first four trials reaches half of the required information size Two new trials are added to the meta analysis A showing no effect and the cumulative Z score now reaches futility and B showing significant benefit of the intervention and the cumulative Z score now reaches significance by crossing both the conventional boundary as well as the O Brian Fleming boundaries Non superiority boundaries need to be used in conjunction with non inferiority boundaries in order to assess for equivalence between two groups Imagine a meta analysis comparing two groups group A and group B If a cumulative Z value falls below the non superiority threshold then group A is not better than group B But it may be worse If the same cumulative Z value also falls above the non inferiority threshold then group A is not worse than group B In this situation it can be concluded that group A and B are equivalent Graphically this area of equivalence is the area within the two boundaries after they cross also called the inner wedge see figure 12 Figure 12 shows an example of a meta analysis that includes all of the components of TSA that have been discussed the required information size two sided significance testing boundaries non superiority futility boundaries
79. ev J Thorlund K Brok J Gluud C Trial sequential analysis may establish when firm evidence is reached in cumulative meta analysis Journal of Clinical Epidemiology 2008 61 64 75 Hu M Cappeleri J Lan KK Applying the law of the iterated logarithm to control type I error in cumulative meta analysis of binary outcomes Clinical Trials 2007 4 329 340 Lan KK Hu M Cappelieri J Applying the law of the iterated logarithm to cumulative meta analysis of a continuous endpoint Statistica Sinica 2003 13 1135 1145 Ioannidis J Lau J Evolution of treatment effects over time empirical insight from recursive cumulative metaanalyses Proc Natl Acad Sci U S A 2001 98 83 1 836 Borm GF Donders AR Updating meta analyses leads to larger type I errors than publication bias J Clin Epidemiol 2009 62 825 830 Brok J Thorlund K Gluud C Wetterslev J Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta analyses Journal of Clinical Epidemiology 2008 61 763 769 Brok J Thorlund K Wetterslev J Gluud C Apparently conclusive meta analyses may be inconclusive Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta analyses International Journal of Epidemiology 2009 38 287 298 Higgins JPT Green S Cochrane Handbook for systematic reviews of interventions version 5 0 0 John Wiley amp Sons 2009
80. ext field and press OK The dos prompt should appear Use the cd change directory command to browse to the folder in which you have unpacked the TSA software For example if you created a folder named TSA within the Program Files folder on your C drive and unpacked the TSA software to this folder you should first change the directory to the 7SA folder in the dos prompt This can be done by typing cd C Program Files TSA no quotes After the directory in the dos prompt has been changed type java jar TSA jar 3 3 1 Why doesn t TSA start If you are having trouble starting the TSA software there are several possible reasons for this Below is a check list to help identify the most likely reasons is the JRE installed on your system is the installed JRE version at least 1 6 did you extract all the files from the ZIP archive did you rename move or delete any of the unpacked files or folders lf a different program other than TSA starts when double clicking the TSA jar file this means that the jar file name extension is not associated to Java JRE If this happens you can either try to start the program manually using a prompt see above or you can try to change the file name association If you are using Windows you can change the association by following the steps below US G2 User Manual for TSA Copenhagen Trial Unit 2011 open an Explorer window e g double click on My Computer and click the
81. f the simulations incorporated time trend bias such as time lag bias and publication bias Such biases have a considerable impact on significance tests in meta analyses Further as previously noted section 2 2 1 Limitations statistical information relies on accurate and reliable estimation of the between trial variance If the between _ Ag User Manual for TSA Copenhagen Trial Unit 2011 trial variance is underestimated for example due to time lag bias the penalised Z statistic will be artificially large For the above reasons it is reasonable to assume that the recommended values in table 2 constitute the very minimum of a range of appropriate choices Appropriate values for dichotomous data meta analyses including only a small number of trials patients and or events are probably higher than those recommended by Hu et al 2 2 7 The B spending function and futility boundaries When a result in a meta analysis is found to be non significant it is important to assess whether this non significance is due to lack of power or whether it is due to underlying equivalency between the interventions The statistical exercise of testing for equivalency i e testing for both non superiority and non inferiority of a given intervention is commonly referred to as futility testing The statistical test thresholds that arise from this exercise are referred to as futility boundaries When a Z curve crosses the futility bou
82. f you are employing conventional confidence intervals you can choose between coverage levels 95 99 99 5 and 99 9 To do so check the Conventional coverage radio button left in the Set Confidence Intervals area click on the drop down box to the right and select your desired coverage Set Confidence Intervals f Conventional coverage oss C oa Spending adjusted Figure 28 Choose you coverage for conventional confidence intervals lf you have already constructed adjusted significance test boundaries using an a spending function see section 2 2 4 and 4 4 1 you will also have the option of obtaining the a spending adjusted confidence interval see section 2 2 5 To do so first click on the a spending adjusted radio button in the Set Contidence Intervals area and subsequently click on the Select button figure 29 69 User Manual for TSA Copenhagen Trial Unit 2011 Set Confidence Intervals Conventional coverage oss Select Figure 29 Select a spending function adjusted confidence intervals A pop up window with a list of your added alpha spending boundaries should appear in the middle of the screen Select which of the alpha spending boundaries the adjustment should be based on and click on the Select button figure 30 Note the cumulative coverage of the alpha spending adjusted confidence intervals will correspond to the alpha level set for the chosen alpha spending functi
83. false positive meta analytic result This section provides basic to intermediate statistical and conceptual descriptions of significance testing in meta analysis and the problems that result from failing to incorporate the strength of evidence and the number of repeated significance tests into the process General criteria for significance testing Conventional significance testing operates with a maximum risk of type error a which also functions as the threshold for when P values are considered evidence of statistical significance P values and Z values are inter changeable in the assessment of statistical significant As mentioned above for every P value threshold a there exists a corresponding Z value threshold Za For example if we desire a maximum two sided type error risk of 5 we should only consider absolute Z values larger than 1 96 as evidence of statistical significance But if we desire a maximum two sided type error of 1 we should only consider absolute Z values larger than 2 58 as evidence of Statistical significance Let Pr X Y denote the probability that the event X occurs given that event Y is true or has occurred let Z denote the absolute value of Z In general we face the challenge of appropriately determining a threshold c that will make the following equations true 3 BR User Manual for TSA Copenhagen Trial Unit 2011 Pr Z 2c Ho is true lt a 2 Pr Z c Ho is true a 3 For the
84. for handling zero event data see section 2 1 4 To select the method you wish to employ for handling zero event data first click on the Method drop box to display the available continuity correction methods and then click on the method you wish to employ figure 27 Set Zero Event Handling Set Zero Event Handling Constant D ethod Constant Constant Reciprocal Indude trials with no events Indude trials with no events Empirical Figure 27 Select continuity correction method by clicking on the Method drop box 68 User Manual for TSA Copenhagen Trial Unit 2011 You also need to set the continuity correction factor In the TSA program the correction factors are derived from sum of the correction factors in the two groups also referred to as the Value For example the sum of the correction factors in the continuity correction used in Review Manager is 1 0 5 0 5 because 0 5 is added to the number of events in both groups To set the sum of two correction factors first click on the Value drop box then select the sum you wish the two correction factors to add up to In addition you have the option of applying continuity correction on trials that have zero events or non events in both arms To do so check the box titled Include trials with no events 4 3 4 Choosing the type of confidence interval TSA provides a number of options for the type of confidence interval you wish to employ figure 28 I
85. ften a meta analysis has been subjected to significance testing as a result of updating For example some meta analyses may include different but highly overlapping data because the inclusion criteria have been modified in connection with updates of a systematic review Other monitoring boundaries such as a set of the monitoring boundaries based on the power family alpha spending function with rho 2 could yield discrepant inferences about statistical significance if for example the monitoring boundaries accounted for 2 previous updates as opposed to 4 Ny O Brien Fleming boundaries gt lt CuFVE Z 1 96 P 0 03 _ Number of 1000 4000 patients Patients Required included information size Figure 7 Example of an inconclusive meta analysis after four cumulative meta analyses Figure 7 shows an example of the use of the O Brien Fleming boundaries In this meta analysis the required information size is 4000 patients but the obtained information is only 1000 patients The final Z value is larger than 1 96 Using the conventional single test threshold this Z value would have led to a conclusion of statistical significance Using the O Brien Fleming boundaries a greater value of Z is required at this information size in 42 User Manual for TSA Copenhagen Trial Unit 2011 order to conclude statistical significance The boundaries are not crossed and the meta analysis is therefore inconclusive ti
86. ge the presentation of the Z curve and your constructed significance tests boundaries Your constructed significance tests and the Z curve will be listed in the white area see figure 49 To change the presentation of one of these first select one of the tests curves on the list and edit according to your preferences In the TSA program you will have the option to edit the colour the line type and the type and size of the icon displayed at each trial or interim analysis as well as the size and font of the test associated with a curve or a test You also have the option to hide a curve test from the graph a oa User Manual for TSA Copenhagen Trial Unit 2011 TSA Trial Sequential Analysis Viewer version 0 9 Beta gt 10 x File Meta analysis Trials TSA Graphs Diversity Tests and boundaries Adjusted Boundaries penalised Tests Layout a TSA is a Two sided graph 7 Curve Discrete Z Curve Cumulative PD ar RF Z Score LONvenvona TSA 2546 Conventional 2 sided 8 Alpha spending 7 N 6 SN z 5 Na w E Se i EH 3 a Color g iia Line type H 3 bo ine type Dotted a f _ Line width zo x 2 Icon None bd 1l Icon size 6 x 1605 Number of rot oa atal Z I patients Font size 12 e Linear scaled a Show Hide E E a neiii a z m S am ii l bd o _ g Trial Distance Scaled X l
87. he estimated intervention effects as statistically significant Empirical evidence also shows that large pooled intervention effects observed in early positive meta analyses tend to dissipate as more evidence is accumulated gt 1 3 Testing for statistical significance before the information size has been reached The aim of a meta analysis is to identify the benefit or harm of an intervention as early and as reliable as possible Therefore meta analyses are commonly updated when new trials are published For example Cochrane systematic review authors are required to update their systematic reviews at least every second year When meta analyses are updated they are repeatedly subjected to the significance testing over time In randomised aay ae User Manual for TSA Copenhagen Trial Unit 2011 clinical trials repeated significance testing on accumulating data is known to inflate the overall risk of type error Simulation studies suggest that if repeated significance testing is done in meta analyses and P values smaller than 0 05 are considered to be evidence of statistical significance then the actual risk of type error will be between 10 and 30 1 31 When decisions made accordingly to implement the intervention as a treatment this means that between 1 and 3 out of 10 treatments decisions are likely inappropriate To deal with this problem one can adjust the thresholds for which results are considered
88. heck the radio button Model Variance Pay ie User Manual for TSA Copenhagen Trial Unit 2011 Based Note that if you have selected the fixed effect model this adjustments factor is always equal to 1 and thus no adjustment is applied The second option is to make a guess estimate of predicted heterogeneity When your meta analysis includes an insufficient number of trials to reliably estimate the adjustment factor you may adjust the required information size for some a priori maximum or plausible anticipated degree of heterogeneity To use this option check the User Defined radio button and type in the maximum anticipated heterogeneity in the input field to the left Here heterogeneity is defined as the percentage of the total variance in the meta analysis which is explained by between trial variation rather than within trial variation Thus a user defined adjustment of 50 for example yields a required information size that allow for reliable inference when approximately half of the total variation among trial in the meta analysis is explained by the between trial variation To set the anticipated event rates and intervention effect for a dichotomous data meta analysis you only need to fill in two of the three fields Relative risk reduction Incidence in Intervention Group and Incidence in Control Group lf you have categorized some of your included trials as low bias risk trials you may use the pool
89. hitehead A A prospectively planned cumulative meta analysis applied to a series of concurrent clinical trials Statistics in Medicine 1997 16 2901 2913 115
90. hoices of the constant A for continuous data meta analysis and Hu et al did the same for dichotomous data meta analysis For continuous data meta analysis Lan et al found that A 2 would generally exhibit good control of the type error when using a desired maximum type error of a 5 for a two sided statistical test i e a 2 5 for each side That is when Z was evaluated based on the conventional criteria for statistical significance i e Z 21 96 means statistical significance at two sided a 5 For dichotomous data meta analysis Hu et al estimated appropriate choices of for different maximum type error levels and different effect measures Their simulation results lead to the recommended values presented in table 2 Max type error corresponding threshold Effect measure a 0 01 c 2 33 a 0 025 c 1 96 a 0 05 c 1 65 Risk difference A 3 ATS K 135 Relative risk A 3 5 A 2 A 2 Odds ratio A 3 5 A 2 A 2 Table 2 Recommended values for penalising Z values with the law of the iterated logarithm These values pertain only to the ranges of study sizes control group event proportion and between trial variation used in the simulations and may therefore not be applicable to all meta analysis scenarios For example the minimum event proportion in the control groups used in the simulations was 0 05 Many important clinical conditions yield control group event proportions lower than 0 05 In addition none o
91. ials and the impact of multiplicity 79794 1 2 Defining strength of evidence information size Meta analyses of randomised trials increase the power and precision of the 13 When all available trials are included estimated intervention effects systematic reviews and meta analyses are considered to be the best available evidence However the best available evidence may not be synonymous with sufficient evidence or strong evidence In a single randomised trial with a binary outcome measure we estimate the number of events and patients needed to allow for reliable statistical inference That is we perform a sample size calculation to ensure that a sufficient number of events and patients are included A similar goal post is needed for a meta analysis This goal post has been referred to as the required meta analysis information size IS or the optimum information Size 1 2 4 6 11 12 14 15 19 23 25 Figure 3 illustrates two typical meta analytic scenarios A and B where the test statistic has stabilised after the required information size has been reached Test A Test B statistic statistic Significant Significant ee me meme u ae a a w a w u u ou ou au a ab b we e e a a t i i i 4 ree e Non significant Non significant IS IS Figure 3 Examples of how the required information size ensures reliable significance tests in two cumulative
92. ided P value can be obtained using the following formula P 2 1 Z where Z denotes the absolute value of the Z value and denotes the cumulative standard normal probability distribution function The P value is the probability of observing a Z value at least as extreme as the one observed due to the play of chance The smaller the P value the smaller is the likelihood that the difference observed between two intervention groups is simply a chance finding and thus the larger is the likelihood that the observed difference was caused by some underlying true treatment effect 2 1 3 Approaches to random effects model meta analysis As explained above the random effects model attempts to include a quantification of the variation across trials The common approach is to sies User Manual for TSA Copenhagen Trial Unit 2011 estimate the between trial variance with some between trial variance estimator 13 The DerSimonian Laird method The between trial variance estimator which has been used most commonly in meta analytic practice and is the only option in The Cochrane Collaboration s Review Manager software is the estimator proposed by DerSimonian and Laird DL The DL estimator takes the form to max 0 Q k 1 S1 S2 S1 Where Q is the Cochrane homogeneity test statistic given by Q w Y ii where S X wf for r 1 2 and where k is the number of trials incl
93. in which cases the estimates of heterogeneity will be larger For meta analyses with an expected small number of trials we suggest that an a priori estimate about the anticipated degree of heterogeneity is made If we let H denote a conceptual estimate of D we can use the following formula in an a priori calculation For example if it is expected that a given meta analysis will contain a mild degree of heterogeneity based on what we know about the clinical topic observed differences between the included trials anticipated differences between current and future and the scope of the review one may choose to define H as 25 In this case the AF would be estimated at 1 33 If a moderate degree of heterogeneity is expected one may choose to define H as 50 and AF would then be estimated at 2 00 If major heterogeneity is expected then H may become 75 and AF would be estimated to 4 00 Because the expected degree of heterogeneity can be difficult to estimate when a meta analysis only includes a few trials we recommend that users of TSA conduct sensitivity analyses for this variable For example one could conceive minimum and maximum realistic or acceptable degrees of heterogeneity for a given meta analysis As an example one could speculate that the minimum plausible degree of statistical heterogeneity would be 20 One could also decide that if the statistical heterogeneity exceeds 60 then subgroup effect measures rather than
94. ined below Project manager KT Principal software application developer JE Co software application developers KT JW JB CG Statistical programmer KT Internal beta testers JB GI JW KT CG Manual authors KT principal GI JW JB JE CG Project supervisors JW and CG User Manual for TSA Copenhagen Trial Unit 2011 Preface This manual provides a guide both theoretical and practical for the use of Copenhagen Trial Unit s Trial Sequential Analysis TSA software Chapter 1 introduces the concepts and rationale chapter 2 provides a technical overview of the implemented methodologies and chapters 3 5 are practical chapters on how to install use and apply the software The TSA software can be downloaded at www ctu dk tsa You are welcome to use it in your analyses and publications of cumulative meta analyses with proper reference to the software and some of our articles describing the methodology In case you need assistance with the TSA software please contact us via email tsa ctu dk User Manual for TSA Copenhagen Trial Unit 2011 1 Concepts and rationale behind trial sequential analysis 1 1 Random error in meta analysis some positive meta analytic findings may be due to the play of chance random error rather than due to some underlying true intervention effect Likewise some neutral or negative non positive meta analytic findings may also represent a chance find
95. ing due to lack of statistical power and precision These two types of errors are commonly known as false positive errors or type errors and false negative errors or type II errors Meta analyses are typically deemed positive or negative on the basis of some Statistical test test statistic communicated with a P value or with the corresponding confidence interval When a meta analysis includes a small number of trials and a small number of patients random errors can cause spurious findings 4 9 11 12 14 15 Conversely when there is a large number of patients and when several trials have confirmed findings of previous trials test statistics and intervention effect estimates will typically converge towards the truth 49 9 11 12 14 15 Figures 1 A and 1 B illustrate examples of such convergence in test statistics In both situations inferences about statistical significance are erroneous at certain early stages but eventually converge to the true side of statistical significance Test A Test B statistic Statistic Significant e oe e e e e e e e e e re oe oe 3e me me ee ee ee ee oee e e e oe oe e Significant Non significant Non significant Number of patients randomised Number of patients randomised Figure 1 Examples of convergence in test statistics as patients are included and followed to an outcome measure e g death in two randomised clinical trials A and B R
96. ing line to the left make up the trial sequential monitoring boundaries which are truncated for the first 4 trials The solid blue line is the Cumulative Z curve 5 3 2 Avoiding early overestimates This same example of atrial fibrillation in CABG can be used to illustrate how overestimates of effect can happen early in the conventional meta analytic process The meta analysis of atrial fibrillation became statistically significant according to the conventional criterion p lt 0 05 after the first trial All except one of the subsequent P values in the cumulative meta analysis were also smaller than 0 05 In fact most subsequent P values were smaller than 0 01 Empirical evidence suggests that pooled effect estimates even when statistically significant are unstable when only a limited number of events and patients have been accrued Insisting that a meta analysis surpasses its required information size may ensure reliable pooled estimates 9 Table 3 shows the evolution of treatment effects over time in this example at the end of each year The pooled relative risk was grossly overestimated in the first two years and supported by P values smaller than 0 01 the conventional 99 confidence intervals precluded 1 00 The following three years the pooled relative risk was overestimated by an absolute risk of at least 10 In 2003 the meta analysis crossed the monitoring boundaries from the information size calculation based o
97. jar is a Java archive containing the Trial Sequential Analysis application RM5Converter jar is a Java archive containing an application for converting trial data presently however for dichotomous outcomes only exported from Review Manager v 5 into the appropriate data format for TSA TEMPLATES TPL contains monitoring boundary templates that you can use when you are performing trial sequential analysis on your meta analysis data The content of the templates file is controlled through the TSA program The folder lib contains various external packages used by the TSA program The folder samples contains TSA files for the examples provided in this manual see chapter 5 To install the program unpack the entire ZIP archive into a folder of your choice on your hard drive No further steps are required 3 3 Starting TSA To start the TSA software double click the TSA jar file a BS User Manual for TSA Copenhagen Trial Unit 2011 Alternatively the TSA software can be started in a prompt To start the TSA software in a prompt first start a prompt browse to the folder in which you have unpacked the TSA software and type java jar TSA jar If you are using the Microsoft Windows operating system you can open a dos prompt by first clicking on the Start button typically lower left corner of the screen then clicking on Run When the Run window pops up type in cmd no quotes in the t
98. l yield spurious inferences if they preclude the null effect This situation is identical to a false positive significance test see section 2 2 4 Similar to adjustment for repeated significance testing the confidence intervals can be adjusted according to the strength of the available information e g the number of patients and the number of statistical evaluations If we let and u denote the lower and upper limit of some na ve confidence interval with coverage 1 a we know that User Manual for TSA Copenhagen Trial Unit 2011 When a meta analysis is subjected to repeated statistical evaluation the repeated naive confidence intervals will not yield the desired coverage Thus we need to establish a series of intervals that will achieve the desired coverage Assume that a meta analysis is subjected to statistical evaluation k times up till the point where it surpasses its required information size Let 4 l2 wey k and Uy Ud Ux denote the lower and upper confidence interval limits for each of the k times the meta analysis was subjected to statistical evaluation To maintain the desired coverage these limits would have to satisfy Pr 1 lt u lt u L SU lt Suw Sp su e And thus any single one of these k intervals say would have to satisfy Pr 1 lt u lt u 21 a It is clear from the above that the a level for each repeated confidence interval cannot exceed the overall maximum a Further the respective a levels f
99. lates Save as template Manage templates Figure 41 Template area where you can load and save add constructed significance tests A pop up window will appear in the middle of TSA program window figure 42 The list of available templates is shown to the left Boundary templates i z Templates available a Information on selected template ey 10 RRR IName 10 RRR Alpha spending 10 RRR IW ALPHA Type Iwo sided 5 type 1 error risk COM Type 1 Error 5 03 Alpha Spending O Brien Fleming Information Axis Sample Size IS Type Estimate Power 80 0 Effect Type Intervention RER User Defined 13 7 Heterogeneity Correction Variance Based Delete selected template Add to Meta Analysis Close Figure 42 Templates window The significance test 10 RRR has been selected and the settings of this test are displayed under Information on selected boundary You can click on a template title to display the available significance tests settings on the right side To load a template significance test for your meta analysis select the template you wish to load and click on the Add to Meta analysis button If you wish to delete one of the available templates permanently select the template you wish to delete and click on the Delete selected Template button 4 4 4 Performing the significance test calculations Once you have added all the significance tes
100. lective reporting of outcomes in randomized trials comparison of protocols to published articles Journal of American Medical Association 2004 291 2457 2465 Chan AW Altman D Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research Canadian Medical Association Journal 2004 171 735 740 Dwan K Alman D Arnaiz J et al Systematic review of the empirical evidence of study publication bias and outcome reporting bias PLoS Medicine 2008 3 e3081 Hrobjartsson A Chan AW Haahr M Gotzsche P Alman D Selective reporting of positive outcomes in randomised trials secondary publication A comparison of protocols with published reports Ugeskr Laeger 2005 167 3189 3191 113 User Manual for TSA Copenhagen Trial Unit 2011 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 Kjaergaard L Villumsen J Gluud C Reported methodological quality and discrepancies between small and large randomized trials in meta analyses Annals of Internal Medicine 2001 135 982 989 Moher D Pham B Jones A et al Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta analyses Lancet 1998 352 609 613 Schulz K Chalmers I Hayes R Altman D Empirical evidence of bias Dimensions of methodological quality associated with estimates of treatment effects in controlled trials Journal of America
101. mber of patients in intervention group X The standard deviation in intervention group X Variance estimate The variance in a fixed effect model The variance in a random effects model The weight assigned to the th trial in a fixed effect model The weight assigned to the th trial in a random effects model The th trial weight as a function of the between trial variance 7 2 2 Upper case letter symbols AF C CFx p E X H Ho 2 l IF S Patients lSEvents S Statistical ISFixea S Random OR P Px p Pr X Pr X Y The heterogeneity adjustment factor The sum of the continuity corrections for two groups The continuity correction for intervention group X The diversity measure used to quantify heterogeneity The expectation of X A conceptual measure of D The null hypothesis The inconsistency measure used to quantify heterogeneity The cumulative statistical information after the j th The cumulative information fraction after the th trial The required number of patients in a meta analysis The required number of events in a meta analysis The required statistical information in a meta analysis The required information size for a fixed effect model The required information size for a random effects model The odds ratio estimate of the th trial The test P value derived from Z The event rate in inte
102. meta analyses A and B 6 User Manual for TSA Copenhagen Trial Unit 2011 A sample size calculation in a single trial is typically based on the expected control event proportion the expected relative risk reduction of the experimental intervention and the desired maximum risk of both type error and type Il error In a meta analysis there is likely heterogeneity across included trial populations interventions and methods Meta analysis sample size considerations need to be adjusted that is increased in order to allow for the variance introduced by this heterogeneity Such adjustments are analogous to adjustments for variation across centres in a multi centre trial 7 Conventional meta analysis methods such as those available in Review Manager v 5 1 do not take into account the amount of the available evidence Instead the reliability of a statistically significant intervention effect is commonly taken for granted irrespective of the accrued number of events and patients Conversely intervention effects that are not statistically significant are commonly not considered reliable Rather it is assumed that more evidence is needed 7 Empirical evidence suggests that intervention effects and P values based on a limited number of events and patients are often not reliable 1 49 9 11 12 29 About 25 of conventional meta analyses that include a small number of events and patients may falsely declare t
103. n Medical Association 1995 273 408 412 Wood L Egger M Gluud LL et al Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes meta epidemioogical study British Medical Journal 2008 336 601 605 Guyatt GH Oxman AD Vist GE et al GRADE an emerging consensus on rating quality of evidence and strength of recommendations BMJ 2008 336 924 926 Reboussin DM DeMets DL Kyungmann K Lan KKG Programs for Computing Group Sequential Boundaries Using the Lan DeMets Method v 2 2009 O Brien PC Fleming TR A multiple testing procedure for clinical trials Biometrics 1979 35 549 556 Pocock SJ When to stop a clinical trial British Medical Journal 1992 305 235 240 Awad T Thorlund K Hauser G Stimac D Mabrouk M Gluud C Peginterferon alpha 2a is associated with higher sustained virological response than peginterferon alfa 2b in chronic hepatitis C Systematic review of randomized trials Hepatology 2010 51 1176 1184 Rigotti N Munafo MR Stead LF Interventions for smoking cessation in hospitalised patients Cochrane Database of Systematic Reviews 2007 Moller CH Penninga L Wetterslev J Steinbruchel DA Gluud C Clinical outcomes in randomized trials of off vs on pump coronary artery bypass surgery systematic review with meta analyses and trial sequential analyses Eur Heart J 2008 29 2601 2616 Brok J Gluud LL Gluud C Meta analysis ribavirin plus interferon vers
104. n the low bias risk estimates 93 User Manual for TSA Copenhagen Trial Unit 2011 and in 2004 the meta analysis surpassed this required information size In 2004 the meta analysis also crossed the monitoring boundaries based on a 20 a priori relative risk reduction Both the conventional and adjusted confidence intervals converged between 2002 and 2004 Total number of Pooled 99 Confidence Interval Year Trials Events Patients Effect Conventional Adjusted 1999 1 55 200 0 24 0 14 to 0 42 0 03 to 7 74 2000 3 74 288 0 39 0 15 to 0 99 0 02 to 7 18 2001 5 143 649 0 57 0 24 to 1 34 0 12 to 2 87 2002 8 204 932 0 52 0 30 to 0 90 0 22 to 1 21 2003 10 285 1168 0 55 0 37 to 0 81 0 35 to 0 85 2003 13 391 1722 0 53 0 35 to 0 79 0 34 to 0 83 2004 19 641 2832 0 61 0 46 to 0 82 2005 20 679 2999 0 63 0 49 to 0 85 2006 25 768 3310 0 67 0 53 to 0 86 2007 27 775 3372 0 67 0 53 to 0 86 First crossing of the boundaries End of the year Table 3 Shows the evolution of pooled effects relative risk estimates conventional and adjusted 99 confidence intervals at the end of each year with respect to the cumulative number of trials events and patients The adjusted 99 confidence intervals are based on alpha spending in relation to the required information size 1964 patients using the relative risk estimate suggested by the trials with low risk of bias This example illustrates why pooled estimates based on a relatively small number
105. n we assume that our pooled estimate of effect is normally distributed as we typically do in meta analysis we form a naive symmetric 95 confidence _44 User Manual for TSA Copenhagen Trial Unit 2011 interval 1 96 se where denotes our estimated meta analysed intervention effect and se 1 denotes its associated standard error However if a meta analysis is subjected to repeated statistical evaluation and thus produces a series of confidence intervals over time the probability that all of these confidence intervals will contain the true overall effect is certainly less than 95 That is if we construct a series of na ve symmetric 1 a confidence intervals i z se 1 the probability that all these confidence intervals will contain the true overall effect is certainly less than 1 a Thus when a meta analysis is subjected to repeated statistical evaluation there is an exaggerated risk that the naive confidence intervals will yield spurious inferences When some underlying true intervention effect exists spurious inferences based on confidence intervals can occur as either of the two scenarios illustrated in figure 10 Spuriously positive Cl j Spuriously negative CI True Null effect effect Figure 10 Example of spuriously positive and spuriously negative confidence interval inferences When there is no intervention effect the confidence intervals wil
106. ncreases The function provides a way to quantify the risk of random error allowed at any given IF in order to ensure that the overall risk of random error after the IS has been reached stays below 5 The monotonically increasing function corresponds to a monotonically decreasing threshold for statistical significance measured by the test statistic Z The a spending function is defined from O to 1 0 being the point where O patients have been randomised and 1 being the point where the accumulated information equals the required information size The a spending function of O is always equal to 0 a 0 0 at this point no information has been accumulated The a spending function of 1 is always equal to a a 71 a at this point all of the required information has been accumulated and the total amount of alpha error is whatever was defined as total acceptable type 1 error overall usually 5 At any point between 0 and 1 for the information fraction at the time of a significance test i IFj the a spending function is equal to the total maximum type error risk that has arisen from the thresholds chosen for all significance tests until and including the i th significance test In other words the a spending function is equal to how much type 1 error has been spent In notation a IF a1 dot aj and thus Pr Z 2 lt a a JR Pr Z 2 Z lt e lt o a0E aQF Pr Z gt Z lt and Z lt e
107. ndaries we can accept that the two interventions do not differ more than the anticipated intervention effect Meta analyses that have already surpassed their required information should have enough power to demonstrate superiority of one intervention over the other For this sub section we will consider only non significant meta analyses that have not surpassed their required information size Further we no longer consider all Z values as absolute Instead we make the distinction of positive Z values indicating that the experimental intervention is superior to the control intervention and negative Z values indicating that the experimental intervention is inferior to the control intervention The following section deals first with non superiority testing followed by non inferiority testing and futility testing in general At any point a meta analysis may yield a Z value that is not statistically significant in favour of the experimental intervention However only when this Z value lies sufficiently below the threshold for statistical significance in 49 User Manual for TSA Copenhagen Trial Unit 2011 favour of the experimental intervention can we be confident that the experimental intervention is not superior to the control To make sense of the above we must first define what we mean by superior and sufficiently below Within the framework of repeated statistical testing the definition of superiority is linked t
108. nical trial eee cccccccesscceceesseeeceessseeeeenseeeees 97 5 6 Other published trial sequential analysis applications ec ccc cceessececeessseeeeessseeees 99 6 1 Effect measures for dichotomous and continuous data meta analysis e 102 6 2 Random effectS ADPrOACHES aniisi n a a a i 103 6 2 1 Formulas for the Biggerstaff Tweedie method 1 0 0 0 ccc cccssecceesssceceessseeceessseeeeens 103 623 TBlali SSG UCIT IVAIY SIS sorar iorn A A E A ee sian OA EE 103 6 3 1 Exaggerated type error due to repeated significance testing cccceeeeeeeees 103 6 3 2 Alternative methods not implemented in the TSA software cccceeeecceeeeeneees 104 7 List of abbreviations and statistical notation ccccceseesseececcccccceceeeeseseecccecesessueeeseecceeeeeeeuaaeeees 108 T e a 2 bers 01g 0 Fa 9 S ernn TT ee 108 f gee feo 1522 notra iO Maa em a E PR EIN A SO EO 108 1 2 1 LOWer Case letter SYMDOIS rirerire a aaa eed ee 108 72 2 J ODEN CaSe OVSn SVIMDOIS srin a E beetaaeeeaede andes 109 TZ MSPOCK IOUT SVIMDOIS 23552 heb aha iets scent dad ares E sateen goon eee ae 110 User Manual for TSA Copenhagen Trial Unit 2011 Disclaimer THE SOFTWARE IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT IN NO EVENT SHALL THE AUTHOR
109. nly test for superiority of the control intervention For binary data meta analysis it should be noted that when the outcome is defined as a positive rather than negative see section 4 1 1 the functions of Upper and Lower are reversed When you have named your conventional boundary and defined the settings press the Add button to add the boundary The a spending boundaries The alpha spending option allows you to add adjusted significance boundaries for the Z curve with the a spending method described in section 2 2 4 Because the a spending method cannot be applied without determining some required meta analysis information size the information size calculations must be defined simultaneously Therefore the a spending boundaries setting window for dichotomous data meta analysis will be different from continuous data meta analysis with respect to the settings for the information size calculation For dichotomous data meta analysis the a spending boundaries setting window that appear when you click on the alpha spending button should similar to the one shown in figure 34 oe User Manual for TSA Copenhagen Trial Unit 2011 Add Dichotomous Alpha spending Boundary Boundary Identifier Name ha Testing Boundary Type C One sided Upper One sidedLower Two sided Type 1 Error Yo a spending Function O Brien Fleming Information Axis Sample Size Event Size Statistical Information Inner
110. ntion effect between the two interventions can be measured as risk difference RD eee User Manual for TSA Copenhagen Trial Unit 2011 relative risk RR or odds ratio OR Intervention effect estimates based on these measures are calculated using the following formulas Rp Ny Mpg z e N Ng _ e n ez Egni e Relative risk ratios and odds ratios will typically be expressed on the log scale because the log transformation induces certain desirable statistical properties such as symmetry and approximate normality Standard errors variances and weights of ratio intervention effects are therefore also obtained on the log scale The formulas for the standard errors of the RD log RR and log OR are provided in appendix 6 1 When the event proportions in the two groups are low rare event data a preferred alternative to the odds ratio is the Peto s odds ratio This odds ratio is calculated with the formula OR pois exp e E e v Where E e is the expected number of events in intervention group A and v is the hypergeometric variance of ea The formulas for E e and v are provided in appendix 6 1 Continuous data effect measures Assume we have k independent trials comparing two interventions intervention A vs intervention B with a continuous outcome Such trials often report the mean response e g mean quality of life score in the two intervention groups m
111. ntion effect is usually expressed as a relative risk reduction RRR When there is limited evidence available about the intervention under investigation one can estimate a clinically relevant intervention effect by using clinical experience and evidence from related areas An example can be found in a paper by Pogue and Yusuf in which the control group event proportion Pc and an a priori RRR were based on experiences from related areas in cardiology Pogue and Yusuf applied information size considerations to two well known meta analyses in cardiology Intravenous Streptokinase in Acute Myocardial Infarction and Intravenous Magnesium in Acute Myocardial Infarction They hypothesized that for most major vascular outcomes such as death it may be realistic to expect 10 mortality in the control group Pogue and Yusuf further considered an example of a theoretical intervention for preventing mortality post myocardial infarction They noted that truly effective treatments for reducing the risk of major cardiovascular events such as death had previously yielded RRRs of 10 15 or at best 20 For any given clinical question a decision needs to be made about what values are appropriate for the Pc and RRR The anticipated proportion of 3s User Manual for TSA Copenhagen Trial Unit 2011 events in the experimental intervention group Pe can then be obtained using the formula Pe Pc 1 RRR Subsequently the hypothesized P
112. o edit a significance test first select the row for the test you wish to edit and then click on the Edit selected button in the Edit area figure 40 Alternatively you can simply double click on the row for the test you wish to edit Edit Edit selected Delete selected Figure 40 The Edit Delete Selected significance test The pop up window with the test s settings will now appear Make your edits and click on the Apply changes button in the lower right corner of the pop up window If you wish to delete a test select the row for the test you wish to delete and press the Delete Selected button in the Edit area Alternatively you can select the row for the test you wish to delete and press the lt Delete gt button on your keyboard 4 4 3 Adding and loading significance test templates The Templates area in the lower left corner of the TSA tab provides you with the option of saving your constructed significance tests and loading previously constructed significance tests figure 41 If you wish to re use a significance test for other meta analyses you can save this in your templates and load it at any other time To save a constructed significance test as a template select the row for the test you wish to save and click on the Save as template button To load a previously saved template first click on the Manage templates button 79 User Manual for TSA Copenhagen Trial Unit 2011 Temp
113. o the underlying assumption made for the required information size When calculating the required information size we assume a priori an intervention effect 6 The magnitude of this effect represents what we believe to be a minimally important difference between the two interventions Ideally the size of 6 should be defined such that anything smaller would be considered clinically or practically unimportant and therefore not worth investigating The value of 6 depends on the context of the study For example a RRR of 10 would usually be considered important if the outcome is mortality but it may not be considered important if the outcome is nausea Before we define what is meant by sufficiently below in the context of repeated statistical testing consider first the situation where the information contained in a meta analysis equals its required information size and where Statistical testing is performed for the first time First let Hs denote the hypothesis that the effect is equal to 6 this is the alternative hypothesis in contrast to the null hypothesis Under the assumption that Hs is true the probability that the meta analysis will be statistically significant with the chosen a level is equal to the chosen power 7 6 When the information size has been reached the probability that the meta analysis will be falsely negative is equal to In this situation our threshold for statistical significance c which satisfies tha
114. ogy 2009 9 Moher D Schulz KF Altman DG The CONSORT statement revised recommendations for improving the quality of reports of parallel group randomised trials Lancet 2001 357 1191 1194 Wetterslev J Thorlund K Brok J Gluud C Estimating required information size by quantifying diversity in a random effects meta analysis BMC Medical Research Methodology 2009 9 Afshari A Wetterslev J Brok J Moller A Antithrombin III in critically ill patients systematic review with meta analysis and trial sequential analysis BMJ 2007 335 1248 1251 Thorlund K Anema A Mills E Interpreting meta analysis according to the adequacy of sample size An example using isoniazid chemoprophylaxis for tuberculosis in purified protein derivative negative HIV infected individuals Clinical Epidemiology 2010 2 57 66 Schulz KF Grimes DA Generation of allocation sequences in randomised trials chance not choice Lancet 2002 359 519 Review Manager RevMan Computer program Version 5 1 Copenhagen The Nordic Cochrane Centre The Cochrane Collaboration 2011 Moher D Tetzlaff J Tricco AC Sampson M Altman DG Epidemiology and reporting characteristics of systematic reviews PLoS Medicine 2010 4 e78 Gehr B Weiss C Porzsolt F The fading of reported effectiveness A meta analysis of randomised controlled trials BMC Medical Research Methodology 2006 6 25 Jennison C Turnbull B Group sequential methods with applications to clinical trials
115. omous data you are required to enter the number of events and total number of patients in the experimental intervention group and the control group figure 21 Add Dichotomous Trial Add Continuous Trial Study Study Wear Year Mean Standard Group Response Deviation Size Event Total Intervention E m m m Control Control Hl Hl Low Bias Risk 7 Low Bias Risk Comment Comment P Add Trial Add Trial Figure 21 Areas where you input the required data when adding a new dichotomous data trial left or continuous data trial right lf you are working with continuous data you are required to enter the mean standard deviation and group size number of patients for the experimental intervention group and the control group figure 21 It is also possible but not necessary to add a comment about the entered data To submit the entered data click on the Add Trial button In the right side of the window you should find four columns Study Bias risk Ignore and Data If you have added trials a list of these trials should appear as in figure 22 The names and publication years of the added trials should appear in the first column from the left in the format year title The assigned bias risks of the respective trials should appear in the second column The bias risk of a trial can either be Low green letters or High red letters The third column gives you the
116. on Select Alpha spending boundary for Confidence intery ls x Select Alpha spending boundary 75 4 ALPHA Upper O Brien Fleming 1 type I error Select Cancel Figure 30 Choose the alpha spending boundaries on which the adjustment should be based Also note that only a spending boundaries that can be calculated and have not been ignored will be included in the list see section 4 4 1 4 4 Applying adjusted significance tests applying TSA TSA currently provides two methods for adjusted significance testing These are the O Brien Fleming a spending method described in section 2 2 4 and the law of the iterated logarithm method described in section 2 2 6 TSA also provides the option to combine the O Brien Fleming method with futility testing as described in section 2 2 7 To apply these methods click on the TSA tab to the right of the Trials tab as shown in figure 31 70 User Manual for TSA Copenhagen Trial Unit 2011 Meta analysis HET rsa DPranhs Diversity Figure 31 Click on the TSA tab when you want apply methods for adjusted significance testing 4 4 1 Adding a significance test In the upper left side of the window you will find the Add area figure 32 which contains the buttons Conventional Test Boundary Alpha spending Boundaries and Law of the Iterated Logarithm When you click on any of these three buttons a new window should appear in the middle of the TSA program window This
117. on alfa 2b for treating chronic hepatitis Ge RM5Converter can read comma separated files csv The first thing you need to do therefore is to convert your RevMan file into a comma separated file Open your RevMan file in Review Manager v 5 in the menu bar select File gt Export gt Data and analyses A pop up window with a check box tree structure will appear figure 16 Check the meta analysis data that you wish to export as a comma separated file and click on the Next button On the following screen check the three first checkboxes Comparison Number Outcome Number and Subgroup Number Then press Finish Note if you click the Next button twice you will be presented with the option of choosing a field delimiter what separates the cells in the data It is important that the field delimiter is a comma this is the default 61 User Manual for TSA Copenhagen Trial Unit 2011 lolx File C 3 Comparisons l C 9 comparison 1 Pegylated interferon plus ribavirin versus non pedylated interferon plus ribavirin 3 3 B T 4 outcomes 1 Sustained virological response 0 H C G lt outcome 2 Liver related morbidity or all cause mortality 0 ee C lt outcome 3 Adverse events 2 El M O lt subgroup gt 1 Adverse events leading to dose reduction 9 2004 Manns 2004 4l Faleh 2004 Bruno 2005 Napoli 2005 PRETTY 2005 Lee 2005 Mangia 2007 3Sjcgren 2008 Roffi subqroup 2 Adverse events le
118. ontrol group event rates and type and type Il errors thus providing a reasonable ballpark interval in which the number of patients need to lie in order to yield a conclusive clinical trial From produced range of sample sizes one would select one primary and let the remaining act as sensitivity sample size power calculations We recommend that information size considerations for meta analysis follow the same construct Low bias risk Pc and RRR estimates could readily be combined with a range of a priori realistic best and worst case intervention effects thus providing a ballpark interval in which the meta analysis information needs to lie in order to yield conclusive meta analytic inferences Limitations The required information size for a meta analysis whether determined as the required number of patients events or statistical information comes with a number of limitations In randomised clinical trials it is reasonable to assume the distribution of prognostic factors in the randomised patients resembles that of the target population In systematic reviews with meta analyses trials are typically included on the basis of a few inclusion criteria that are decided upon in the protocol stage of the systematic review Because inclusion and exclusion criteria in clinical trials are almost never identical and because trials typically vary in sample sizes meta analysts and systematic review authors are unlikely to have control over the di
119. or each of the repeated confidence intervals should sum to the overall maximum a Thus by controlling the overall a level we can control the overall coverage The framework for controlling the overall a level has already been developed in the previous section 2 2 4 and is easily applied to repeated confidence intervals Na ve confidence intervals are obtained using the formula i z _ 8e 1 because we know that z lt se lt Zan with approximately 1 a probability under the null hypothesis and hence Zar SL SZ an where Z denotes the Z value for the statistical significance test By replacing Za 2 ANd Z1 a 2 by the thresholds that constitute the statistical monitoring boundaries C4 Co Ck and isolating for we have constructed a simple expression for repeated confidence intervals which will maintain good control of the coverage For any single one of the k confidence intervals say the expression for the confidence interval is 46 User Manual for TSA Copenhagen Trial Unit 2011 fitc se ft And we have Pr fi c se t lt u lt A se A d e se fi us ite se i gt 1 a All of the above easily generalises to one sided confidence intervals The TSA software provides the option of calculating the confidence interval for the last of a series of statistical evaluations see chapter 4 2 2 6 The law of the iterated logarithm Another solution to the problem of repeated significance
120. or the DL and BT models see section 2 1 3 In the lower right corner there is an option to choose the number of decimal points that all quantities should be displayed wtih Click on the drop down window to select the number of decimal points TSA Trial Sequential Analysis Viewer version 0 9 Beta 5 x File 1995 Test 1 10 0 100 0 15 0 102 0 7 43 10 33 0 1 14 43 4 5 14 45 10 14 12 05 1998 Test 2 27 0 150 0 22 0 148 0 141 07 15 4 0 15 22 19 6 92 22 19 0 22 21 82 1999 Trial 3 7 0 80 0 14 0 85 0 6 79 19 44 0 09 12 08 3 77 12 11 0 12 9 76 2001 Trial 4 28 0 340 0 40 0 320 0 20 61 28 66 0 29 24 65 7 69 24 64 0 25 26 1 2004 Trial 5 25 0 140 0 52 0 140 0 26 0 36 16 0 36 26 64 8 3 26 61 0 27 30 27 Diversity Type ee DL Random SJ Decimals 2 v Not applicable Not applicable Not applicable Not applicable 0 54 2 16 Figure 55 Diversity tab _87 User Manual for TSA Copenhagen Trial Unit 2011 5 TSA example applications 5 1 Datasets To illustrate the TSA applications we use data from several published systematic reviews Some of the analyses and applications presented in this chapter are our own modifications and additions to those that can be found in the original publication 5 2 Avoiding false positives In this example w
121. otizing entercolitis it is time to change practice Pediatrics 2010 125 1068 1070 Whitfield K Rambaldi A Wetterslev J Gluud C Pentoxifylline for alcoholic hepatitis C Cochrane Database of Systematic Reviews 2009 Whitlock R Chan S Devereaux PJ Clinical benefit of steroid use in patients undergoing cardiopulmonary bypass a meta analysis of randomized trials European Heart Journal 2008 29 2592 2600 Bangalore S Kumar S Wetterslev J Messerli FH Angiotensin receptor blockers and risk of myocardial infarction meta analyses and trial sequential analyses of 147 020 patients from randomised trials BMJ 2011 342 d2234 Bangalore S Kumar S Wetterslev J et al Carotid artery stenting vs carotid endarterectomy meta analysis and diversity adjusted trial sequential analysis of randomized trials Arch Neurol 20113 68 172 184 Bangalore S Kumar S Kjeldsen SE et al Antihypertensive drugs and risk of cancer network meta analyses and trial sequential analyses of 324 168 participants from randomised trials Lancet Oncol 2011 12 65 82 Bollen C Uiterwaal C Vught A van der Tweel I Sequential meta analysis of past clinical trials to determine the use of a new trial Epidemiology 2006 17 644 649 Higgins JPT Whitehead A Simmonds M Sequential methods for random effects meta analysis Stat Med 2011 30 903 921 van der Tweel I Bollen C Sequential meta analysis an efficient decision making tool Clinical Trials 2010 7 136 146 W
122. performed over time one including trial 1 and 2 one including trials 1 to 4 and one including trials 1 to 5 Note that the last trial on the list should always be checked as this represents the significance test you are employing on all included trials Interim analyses 1995 Trial 1 I 1998 rial 2 11 999 Trial 3 fe 2004 Trial 5 Figure 36 Alpha spending boundary setting pop up window for continuous data meta analysis that appears when clicking on the alpha spending button In some cases you may wish to check or uncheck all trials for previous significance tests Click on the Select none button in the bottom of the Interim analyses area to uncheck all trials or click on the Select all button to check all trials figure 37 In addition you have the option to inverse the selection of interim looks Select all Select none Inverse selection Figure 37 Check or uncheck all trials for previous significance tests ags User Manual for TSA Copenhagen Trial Unit 2011 The law of the iterated logarithm penalised Z curve The law of the Iterated logarithm option allows you to perform adjusted significance testing by penalising the Z curve with the methods described in section 2 2 6 When you click on the Law of Iterated Logarithm button a window like the one shown in figure 38 should appear You will need to give your Z curve penalisation a name e g 5 symmetric LIL define whether your
123. re illustrated in the right graph of figure 65 The triangular test boundaries are statistically constructed to yield the minimum possible risk of committing an error either a type error or type II error This emphasis on minimising both types of error skews this technique towards favouring total risk of error over risk of type error In the context of medical research conventional theory does not support this balance prevention of alpha error has always been considered more important The performance of the Whitehead triangular test applied in meta analysis has been explored in a simulation study where the method was found to exhibit poor control of the maximum type error in heterogeneous meta analyses The results of this study suggested that the more heterogeneous a meta analysis data set is the worse the triangular test exhibits control of the type error To date the literature contains one example of the Whitehead triangular test being applied to meta analysis comparing death or chronic lung disease after high frequency ventilation with conventional mechanical ventilation in the treatment of preterm newborns In this example the meta analysis demonstrated no difference between the two interventions as the cumulative score statistic crossed the futility boundaries Stochastic curtailment is another method for controlling the risk of false positives and false negatives When applied to meta analysis thi
124. re the experimental intervention is statistically significantly superior to the control intervention and too many trials may have been conducted B TSA provides a technique for finding a conclusion of no effect as early as possible Futility boundaries which were originally developed for interim analysis in randomised clinical trials are constructed and used to provide a threshold for no effect User Manual for TSA Copenhagen Trial Unit 2011 If the experimental intervention is truly superior to the control intervention one would expect the test statistic to fluctuate around some upward sloping Straight line eventually yielding statistical significance when the meta analysis is sufficiently powered If a meta analysis of a truly effective experimental intervention includes only a small number of events and patients the likelihood of obtaining a statistically significant result is low due to lack of power However as more evidence is accumulated the risk of getting a chance negative finding decreases Futility boundaries are a set of thresholds that reflect the uncertainty of obtaining a chance negative finding in relation to the strength of the available evidence e g the accumulated number of patients Above the thresholds the test statistic may not have yielded statistical significance due to lack of power but there is still a chance that a statistically significant effect will be found before the meta analysi
125. resholds for statistical significance the method constructs adjusted thresholds for non superiority and non inferiority or no difference Together adjusted non superiority and non inferiority boundaries make up what is referred to as futility boundaries or ie User Manual for TSA Copenhagen Trial Unit 2011 inner wedge boundaries Sections 2 2 7 provides a description of the underlying methodology and theoretical considerations for this method As previously described an alternative approach to the alteration of thresholds is to penalise the test statistic itself The method for penalising the employed statistical tests is a relatively new approach which builds on theorems from advanced probability theory In particular the technique uses the theorem known as the law of the iterated logarithm Sections 2 2 2 and 2 2 6 provide a description of the underlying methodology and theoretical considerations for this method 2 2 1 The information size required for a conclusive meta analysis Determining the required information size e g the required number of patients for a conclusive and reliable meta analysis is a prerequisite for constructing adjusted thresholds for statistical significance using TSA 1746 11 12 The levels of the thresholds must be constructed in accordance 1 2 4 6 1112 The statistical methodology underlying with the strength of evidence TSA is based on the assumption that data will accumula
126. response is a good thing e g platelet count The TSA software requires the designation of the outcome as negative or positive to determine which intervention arm the results favour After creating your new meta analysis a number of options should appear in the left side of the starting window These options will be described in section 4 3 Defining your meta analysis settings 59 User Manual for TSA Copenhagen Trial Unit 2011 TSA Trial Sequential Analysis Viewer version 0 9 Beta 10 x File heeesessseessseeeesesseeesessseeeesed High perioperative oxygen fraction for SSI 80 vs 30 35 Set Effect Measure and Model Effect Measure Relative Risk v Model Random Effects DL Set Zero Event Handling Method Empirical v Value 0 01 Meta analysis Summary Indude trials with no events Pooled effect 0 89 Conventiona CI 0 65 to 1 23 Set Confidence Intervals P value 0 4822 Conventional coverage 95 Heterogeneity Q 18 41 C a spending adjusted 95 CI Heterogeneity Q P value 0 0053 Inconsistency 13 67 Diversity D2 79 Decimals 2 Figure 15 Starting window after a new meta analysis has been created and data have been entered A box tited Meta analysis Summary should appear in the middle of the window 4 1 2 Saving a TSA file and opening an existing TSA file If you wish to save your work go to the menu bar and sel
127. rvention group X The average event rates of the two treatment groups The probability that some event X occurs The probability that some event X given the event Y occured 109 User Manual for TSA Copenhagen Trial Unit 2011 Q The Cochran homogeneity test statistic R The randomisation ratio RD The risk difference estimate of the th trial RR The relative risk estimate of the th trial S The sum of trial weights to the r th power SE X The standard error of X Var X The variance of X Z The test statistic for whether there exists an intervention effect Zi The Z value from the meta analysis including the first trials L1 0 2 The 1 a 2 th percentile of the standard normal distribution Z1 p The 1 B th percentile of the standard normal distribution Yi The observed intervention effect in the i th trial 7 2 3 Greek letter symbols q The maximum risk of type error a t The cumulative type error risk as a function of time B The maximum risk of type II error B t The cumulative type II error risk as a function of time The a priori estimate of an anticipated intervention effect F The anticipated intervention effect in a fixed effect model R The anticipated intervention effect in a random effects model Xr A constant to ensure control of a when penalising Z Lj The underlying true intervention effect of the th tri
128. s surpasses the IS Below the threshold the test statistic is so low that the likelihood of a significantly significant effect being found becomes negligible In the latter case further randomisation of patients is futile the intervention does not possess the postulated effect Figure 5 A illustrates an example where the experimental intervention is not superior to the control intervention The test statistic crosses the futility boundaries the upward sloping concave curve before the required information size is surpassed Figure 5 B illustrates an example where the experimental intervention is statistically significantly superior to the control intervention In this example the test statistic stays above the futility curve because there is an underlying effect and eventually yields statistical significance 1 5 Summary Trial sequential analysis TSA is a methodology that uses a combination of techniques The evidence required is quantified providing a value for the required IS The thresholds for statistical significance are adjusted and these modifications are done according to the quantified strength of evidence and the impact of multiplicity Thresholds for futility can also be constructed using a similar statistical framework 10 User Manual for TSA Copenhagen Trial Unit 2011 In summary TSA can provide an IS a threshold for a statistically significant treatment effect and the threshold for futility Con
129. s method concentrates on predicting what the outcome will be once a meta analysis surpasses its required information size More specifically stochastic 106 User Manual for TSA Copenhagen Trial Unit 2011 curtailment is a method for calculating the likelihood that the current trend of the data will reverse before surpassing the required information size When the probability of such a reversal is sufficiently small a meta analysis may be considered conclusive Two conditional probabilities can be calculated First if the current trend in the data is suggesting that the experimental intervention is effective stochastic curtailment may be used to calculate the probability of rejecting the null hypothesis when the meta analysis surpassed the required information size If this conditional probability is sufficiently high the meta analysis can be considered conclusive Similarly if the current data is suggesting a lack of trend stochastic curtailment can be utilised to calculate the probability of failing to reject the null hypothesis once the meta analysis surpasses its required information size Again if this conditional probability is sufficiently high the meta analysis can be considered conclusive Stochastic curtailment may be a valuable tool to assist decision making from formal significance testing methods However because most meta analyses are subject to some time trend bias the conditional probability of a trend reversal is
130. stribution of prognostic factors Even when some systematic review inclusion criteria are altered for an update authors will not be able to accurately predict the distribution of prognostic factors across newly published trials Baseline prognostic factors can have a Be es User Manual for TSA Copenhagen Trial Unit 2011 considerable impact on incidence rates in a control group In this situation it may be appropriate to make an a priori attempt at quantifying the difference between the baseline incidence in the meta analysis population and that in the target population and perform post hoc sensitivity analyses if necessary Minimally important comparative intervention effects also known as minimally important differences may not always be similar across the included trials For example if the investigated patient populations across trials experience different risks of adverse events the minimally important difference may also differ across trials This variation is the result of clinical intent For any medical intervention the chance of benefit needs to outweigh any increased risk of harm A population with greater risk of harm will need a greater chance of benefit to make a treatment worthwhile When minimally important differences vary across trials information size considerations may still be sensible However it is important to remember that inference drawn about the conclusiveness of a meta analysis can only be generalized to th
131. t Pr Z 2c Ho is true lt a implicitly becomes our threshold for non superiority because c also satisfies Pr Z lt c Hs is true lt B 50 User Manual for TSA Copenhagen Trial Unit 2011 When repeated statistical testing occurs before a meta analysis surpasses its required information size it is also possible to test for non superiority This testing can be done by defining thresholds that under the alternative hypothesis do not result in an inflation of the total risk of type II error For example if we test for non superiority two times we need to find thresholds C and Co for the emerging two subsequent Z values Z and Zz Pr Z lt 4 rZ lt c lt In this situation Z4 values smaller than c and Z2 values smaller than C2 will be considered sufficiently below the threshold for statistical significance to justify the conclusion of non superiority In a more general context where we might test for non superiority k times we would need to find thresholds c4 Ck which will satisfy Pr Z lt 6 oZ lt c or orZ lt c lt B under the alternative hypothesis Hs This is equivalent to finding k maximum type Il error risks By Bx that sum to 6 and where Pr Z lt q lt 6 Pr Z lt o Z 2 4 lt Pr Z lt c Z 2 4 andZ gt lt lt Pr Z lt C Z gt c and and Z 2c lt B under the alternative hypothesis This desire to control the type Il error in the context of repeat
132. t is therefore possible to define known group sizes between each interim look In meta analysis an interim look at the data occurs when there is an update adding data from new clinical trials Updates in meta analysis occur at an arbitrary pace are seldom regular and the number of added patients is varied and unpredictable The methods proposed by Armitage and Pocock are therefore inapplicable for meta analysis Lan and DeMets extended the methodology proposed by Armitage and Pocock allowing for flexible unplanned interim analyses Lan and DeMets intended this methodology for repeated significance testing in a single randomised trial Because of the flexibility of the timing of interim looks this methodology is applicable to meta analysis The Lan and DeMets approach is therefore the methodology used in TSA it involves construction of monitoring boundaries that facilitate the definition of sensible thresholds for statistical significance in meta analysis Similarly futility boundaries can be constructed facilitating the definition of sensible thresholds for futility in meta analysis Sections 2 2 1 to 2 2 5 provide a description of the underlying methodology and theoretical considerations for these methods The methods for controlling for type Il error are an extension of the Lan DeMets methodology that allows for non superiority and non inferiority testing That is instead of constructing adjusted th
133. t produces conservative boundaries at early stages where only limited amount of data has been accumulated and more lenient boundaries as more data are accumulated The O Brien Fleming boundaries have been recommended by methodological experts as the preferred choice in most randomised clinical trials where repeated significance testing on accumulating data is performed In meta analysis where the risk of random error and time trend biases is of particular concern at early stages i e in meta analyses including a small number of patients and events the O Brien Fleming boundaries have been the preferred choice as well 1746 11 12 There are two reasons for this preference First if the heterogeneity adjustment of the required information size is based on a reasonable a priori estimate of the anticipated degree of heterogeneity the O Brien Fleming 4 User Manual for TSA Copenhagen Trial Unit 2011 boundaries will naturally account for the degree of fluctuations that the meta analytic inferences will incur due to random error and heterogeneity Second as long as subsequent significance tests are performed at a reasonable distance on the information axis e g at least 1 of the required information size apart the O Brien Fleming boundaries remain relatively unaffected by the number of previous significance tests This second property is desirable in the setting of meta analysis because it is not always clear how o
134. te until the required information size is surpassed For further explanation on this assumption please refer to earlier methodological papers on this issue 16 1 30 43 44 Conventional information size considerations It has been argued that the sample size required for a conclusive and reliable meta analysis should be at least as large as the sample size required to detect a realistic intervention effect in a large reasonably powered 17461112 in line with this construct the minimum required information tria size number of patients in a meta analysis can be derived using the well known formula ISpatients 2 Zir Z 2 0 Le 1 where a is the desired maximum risk of obtaining a false positive result type error and B is the desired maximum risk of obtaining a false negative result type Il error and where Z7 2 and Z1 g are the 7 a 2 and 1 6B standard OAS User Manual for TSA Copenhagen Trial Unit 2011 means that the information size is constructed assuming two sided statistical testing For binary data 6 Pc Pe denotes an a priori estimate for a realistic or minimally important intervention effect Pc and Pe being the proportion with an outcome in the control group and the in the intervention group respectively where of P 1 P which is the associated variance and assuming P Pc Pe 2 i e that the intervention and control groups are equal in size For continuous data 6 denotes an
135. ted intervention is available from a number of trials with low risk of bias it would be appropriate to base an a priori anticipated intervention effect on a meta analysis of these trials However meta analytic situations that call for information size calculations will often occur when the evidence is sparse Even if a number of trials with low risk of bias are available for approximating an anticipated realistic intervention effect the pooled estimate from these trials may still be subject to considerable random error time lag bias and publication bias An a priori anticipated intervention effect based on the pooled effect estimate from a meta analysis of trials with low risk of bias is therefore only reliable to the extent that this meta analysis Be es User Manual for TSA Copenhagen Trial Unit 2011 can be considered free of large random errors Furthermore it is only valid to the extent it can be considered free of time lag bias and publication bias It is not possible to recommend one technique for defining intervention effects for information size calculations Rather information size considerations should be based on ranges of plausible control group event proportions intervention effects and suitable type and type II errors Adequate sample size considerations for a single clinical trial do not just amount to one single number Instead a range of plausible sample sizes are produced from a range plausible treatment effects c
136. test is two sided symmetric or one sided what your overall maximum type error will be and set your penalisation parameter see section 2 2 6 and table 2 For one sided tests the Upper one sided test will only test for superiority of the experimental intervention whereas the Lower will only test for superiority of the control intervention For binary data meta analysis it should be noted that when the outcome is defined as a positive rather than a negative outcome see section 4 1 1 the functions of Upper and Lower are reversed ES add Law of the Iterated Logarithm E x Boundary Identifier Mame Boundary Settings Boundary Type C One sided Upper One sidedLower Two sided Type 1Eror To Penalty A penalty Figure 38 Law of the Iterated logarithm penalisation setting pop up window that appears when clicking on the Law of Iterated Logarithm button 4 4 2 Editing and deleting a significance test Whenever a significance test is added it should appear in the middle of the screen Each significance test you add will be represented by a row as shown in figure 39 78 User Manual for TSA Copenhagen Trial Unit 2011 Identifier TSA ALPHA FE Alpha spending Conventional 2 sded COW a a e LIL 5 Law of Iterated Logarithm Figure 39 List of added significance tests T
137. the button in the bottom of the window titled Create TSA file s A pop up window allowing you to review the names of your interventions outcomes and subgroups will appear Once you have reviewed your selections click on the button Create TSA file s and save your selected meta analyses as TSA files in the specified folder 4 2 Adding editing and deleting trials Right below the menu bar in the TSA program you will find five tabs Meta analysis Trials TSA Graphs and Diversity To add edit or delete any trials in your meta analysis first select the Trials tab figure 20 MESHES mat P rans oversty Figure 20 Click on the Trials tab when you want to add edit or delete trials in your meta analysis In the left side of the window in the Trials tab there should be three areas Add Dichotomous Continuous Trial Edit Delete Trial and Ignore Trials 4 2 1 Adding trials To add a new trial fill in the input fields in the Add Dichotomous Continuous Trial area Regardless the type of data you are meta analysing you are required to provide some name or title for the study in the Study input field 64 User Manual for TSA Copenhagen Trial Unit 2011 typically the study acronym or the last name of the first author You also need to provide the year that the study was published in the Year input field You have the option to check the trial as a low bias risk trial lf you are working with dichot
138. tion is equal to how much type II error has been spent In notation B IF 61 Bot 6 For the same reasons described in section 2 2 4 the O Brien Fleming function may also constitute the optimal choice for the beta spending function In TSA v 0 8 the only available B spending function is the O Brien Fleming spending function Figure 11 shows an example of a meta analysis including both repeated non superiority and significance testing In this meta analysis the required information size is 4000 patients At 2000 patients the meta analysis is inconclusive because it has not yet crossed the upper boundary for Statistical significance or the lower boundaries for non superiority The dashed extensions of the Z curve illustrate examples of how the meta analysis could become conclusive at 3000 patients In example A the Z curve crosses the non superiority boundaries the lower boundaries in which case it would be inferred that the experimental intervention is not superior to the control intervention In example B the Z curve crosses the O Brien Fleming significance boundaries for superiority in pee User Manual for TSA Copenhagen Trial Unit 2011 which case it would be inferred that the experimental intervention is superior to the control intervention O Brien Fieming boundaries Cumulative Z Value mt Z curve Z 1 96 P 0 05 a gt A a A Number of V i 2000 4000 panes Patients included R
139. toring boundaries or the futility boundaries If a future clinical trial were to make the meta analysis conclusive with a positive result we assumed that the trials would have the same control group event proportion and intervention effect as hypothesized in the information size considerations That is we assumed a trial would have a 5 control group event proportion and yield a 25 relative risk reduction i e the trial would have a 3 75 intervention group event proportion 98 User Manual for TSA Copenhagen Trial Unit 2011 Cwnulatre Z acore Moderate evidence 10505 E bo A p B E E mo f D aI 6911 i Number of a O patients E en ie cal Linear scalect doa 3 ius kaa i pee ag mG il a 8 d pg 5 fy ra Fi ae ra 7 a F Pa 5 Fa 1 Figure 64 Prospective trial sequential analysis of isoniazid vs control for preventing tuberculosis To the left the red inward sloping dashed lines make up the trial sequential monitoring boundaries To the right the red outward sloping dashed lines make up the futility region The solid blue line is the cumulative Z curve The last line on the Z curve represents an imagined trial that makes the meta analysis conclude that isoniazid is not able to prevent tuberculosis If a future clinical trial were to make the meta analysis conclusive with a futile result we assumed that the intervention group event proportion would also be 5 i e
140. ts you wish to employ you need the TSA program to perform the necessary calculations To achieve this click 80 User Manual for TSA Copenhagen Trial Unit 2011 on the Perform calculations button in the Calculations area under the Edit area Depending on how many significance test you have added the TSA program might take a few seconds to complete the calculations O Brien Fleming a spending type boundaries with many interim analyses can take 5 10 seconds per set of boundaries to compute Calculations Perform calculations Figure 43 Perform calculations button In some instances there is such a small relative increase in information between two interim analyses that the numerical calculations numerical integration of extremely small tail probabilities for the a spending boundaries break down For example if the required information size is 20 000 patients and the interim analyses are performed after each trial adding a new trial with 40 patients would only provide a 0 2 increment in the cumulative information fraction To avoid breakdowns in the calculations the TSA program automatically removes un checks interim analyses that correspond to an information fraction increment of 1 or smaller When this happens a window will automatically pop up in the middle of the TSA program window to inform which interim analyses were removed figure 44 The data of these trials are however retained in your TSA meta anal
141. two interventions is described as statistically significant By definition a P value is the probability of finding the observed difference or one more extreme if the null hypothesis was true In practice the P value is the value that we use to assess Statistical significance The P value is obtained from the Z value see section 2 1 2 for the mathematical details these two measurements represent two different ways of communicating the same information and they are inter changeable For example a two sided P value smaller than 5 is the same thing as an absolute Z value larger than 1 96 and vice versa i34 User Manual for TSA Copenhagen Trial Unit 2011 Every time a meta analysis is updated a new Z value is calculated A series of consecutive Z values therefore emanates from a series of meta analysis updates To inspect the evolution of significance tests the series of Z values can be plotted with respect to the accumulated information accumulated patients events or statistical information thus producing a curve which is commonly referred to as the Z curve 4911 12 2 2 3 Problems with significance testing in meta analysis As mentioned in chapter 1 conventional significance testing in meta analysis fails to relate observed test statistics and P values to the strength of the available evidence and to the number of repeated significance tests 14 11 12 The consequence of this omission is an increased risk of obtaining a
142. uded in the meta analysis Because the DL estimator is prone to underestimate the between trial 33 40 we have included two alternative random effects model variance approaches the Sidik and Jonkman SJ and the Biggerstaff and Tweedie BT methods in the TSA software The Sidik Jonkman SJ method The SJ random effects model uses a simple non iterative estimator of the between trial variance based on a re parametrisation of the total variance of the observed intervention effect estimates Y It is given by the expression ts 5 vi Y uo k 1 where v 7 1 r oflt and to is an initial estimate of the between trial variance which can be defined for example as Tto X Y Hi SK Uuw being the unweighted mean of the observed trial effect estimates and uo being the weighted random effects estimate using t as the estimate for the a he User Manual for TSA Copenhagen Trial Unit 2011 between trial variance Simulation studies have demonstrated that the SJ estimator provides less downward biased estimates of the between trial variance than the DL estimator That is the SJ method is less likely to under estimate the heterogeneity between trials This is particularly the case for meta analysis data that incur moderate or substantial heterogeneity Confidence intervals based on the SJ estimator have coverage close to the desired level e g 95 confidence intervals
143. uential monitoring boundaries One solution to the problem outlined in section 2 2 3 is to adjust the thresholds for the Z values allowing the type error risk to be restored to the desired maximum risk In the two tests example we would thus need to find two thresholds c and Co for which Pr Z 2 or Z 20 lt a is satisfied under the null hypothesis This is equivalent to finding two maximum type error risks a and a2 that sum to a and where Pr Z q lt a Pr Z gt Z lt q lt a oo User Manual for TSA Copenhagen Trial Unit 2011 under the null hypothesis In the general situation where repeated significance testing is employed k times i e where one initial meta analysis and k 7 updates are performed we would need to find thresholds c4 Ck for each of the k significance tests that will ensure Pr Z 2 lt or Ziz Or OF Z 2 lt a under the null hypothesis This is equivalent to finding k maximum type error risks a4 Ax that sum to a and where Pr Z e Z lt a lt a Pr Z gt lt q and Z lt e lt o Pr Z gt q A and and Zalo lt a under the null hypothesis The collation of thresholds for the Z curve is referred to as monitoring boundaries or group sequential monitoring boundaries a series of boundaries applied to sequence of tests on cumulative groups of patients randomised in a clinical trial In meta analysis such boun
144. us interferon montherapy for chonic hepatitis C an updated Cochrane review Alimentary Pharmacology and Therapeutics 2010 32 840 850 Brok J Gluud C Ribavirin monotherapy for hepatitis C Cochrane Database of Systematic Reviews 2010 Issue 1 Ghandi GYMM Flynn DN et al Effect of perioperative insulin infusion on surgical morbidity and mortality systematic review and meta analysis of randomized trials Mayo Clinique Procedings 2008 83 418 430 Knorr U Vinberg M Kessing LV Wetterslev J Salivary cortisol in depressed patients versus control persons a systematic review and metaQanalysis Psychoneuroendocrinology 2010 35 9 1275 86 Nielsen N Friberg H Gluud C Herlitz J Wetterslev J Hypothermia after cardiac arrest should be further evaluated a systematic review of randomised trials with meta analysis and trial sequential analysis International Journal of Cardiology 2010 Rambaldi A Saconato HH Christensen E Thorlund K Wetterslev J Gluud C Systematic review of glucocorticosteroids for alcoholic hepatitis a Cochrane Hepato Biliary Group systematic review with meta analysis and trial sequential analysis of randomised clinical trials Alimentary Pharmacology and Therapeutics 2008 27 1167 1178 114 User Manual for TSA Copenhagen Trial Unit 2011 72 73 74 75 76 77 78 79 80 81 Tarnow Mordi WO Wilkinson D Trivedi D Brok J Probiotics reduce all cause mortality and necr
145. very likely to be biased as well 107 User Manual for TSA Copenhagen Trial Unit 2011 7 List of abbreviations and statistical notation The following chapter provides a guide to the abbreviations notation and terminology used in this manual In some cases these definitions will vary from other sources Our intention is to provide the reader with a guide for how these terms were used in this manual 7 1 General abbreviations AF Adjustment Factor BT Biggersaff Tweedie Cl Confidence Interval D Diversity DL DerSimonian Laird I Inconsistency IF Information Fraction IS Information Size JRE Java Runtime Environment MD Mean Difference OIS Optimal Information Size OR Odds Ratio RCT Randomised Controlled Trial RD Risk Difference RR Relative Risk RRR Relative Risk Reduction SJ Sidik Jonkman SMD Standardised Mean Difference TSA Trial Sequential Analysis 7 2 Statistical notation 7 2 1 Lower case letter symbols C The statistical significance threshold with respect Z Cj The adjusted threshold for Z under repeated testing ex The number of events in intervention group X 108 User Manual for TSA Copenhagen Trial Unit 2011 fp t k mx nx sdy V VF VR Wi w Wi i t The probability distribution for the DerSimonian Laird estimator The number of trials in a meta analysis The mean response in intervention group X The nu
146. will contain the true effect in approximately 95 of all meta analyses In contrast the commonly reported coverage of confidence intervals based on the DL estimator is often below the desired level For example many simulation studies that have investigated the coverage of DL based 95 confidence intervals have found an actual coverage of 80 92 The size of these confidence intervals is equivalent to a false positive proportion of 8 to 20 which is clearly larger than the conventionally accepted 5 The Biggerstaff Tweedie method Because most meta analyses contain only a limited number of trials between trial variance estimation is often subject to random error Incorporating the uncertainty of estimating the between trial variance in the random effects model may therefore be warranted Biggerstaff and Tweedie BT proposed a method to achieve such incorporation They derived an approximate probability distribution fp for the DL estimate of 1 Defining the trial weights as w t of t where tis a variable that can assume all possible values for 7f they utilised fp and obtained trials weights that take the uncertainty of estimating 7 into account This generally creates a weighting scheme which relative to the DL approach attributes more weight to larger trials and less weight to smaller trials Biggerstaff and Tweedie also proposed an adjusted formula for the variance of the meta analysed intervention
147. window should contain a number of fields which will allow you to define the settings for the type of significance test you wish to apply Alpha spending Boundaries Law of the Iterated Logarithm Figure 32 Click on one of the buttons to add a new significance test The conventional significance boundary The Conventional option allows you to add a boundary for the Z curve which corresponds to a single significance test with some maximum type error risk a For example a conventional boundary for a two sided a 5 single significance test will produce two horizontal lines at 1 96 and 1 96 When you click on the Conventional button a window similar to the one shown in figure 33 should appear oF ie User Manual for TSA Copenhagen Trial Unit 2011 tS Add Conventional Test E 3 x Boundary Identifier Name Boundary Settings Boundary Type One sided Upper One sidedLower Two sided Type 1 Error To Add Cancel Figure 33 Conventional Test setting pop up window that appears when clicking on the Conventional Test Boundary button You will need to give your conventional test a name e g single test 5 threshold define whether your test is two sided symmetric or one sided and what your overall single test maximum type error will be For one sided tests the Upper one sided test will only test for superiority of the experimental intervention whereas the Lower will o
148. y optimal thresholds is given by the expression o IF 2 29 Z VIF where is the standard normal cumulative distribution function The type of boundaries produced by this a spending function were first proposed for equal increments of IF by O Brien and Fleming Lan and DeMets later proposed the above a spending function to allow for flexible increments in IF For this reason the above a spending function is typically referred to as the Lan DeMets implementation of the O Brien Fleming a spending function Often the monitoring boundaries produced by this alpha spending function are simply referred to as the Lan DeMets monitoring boundaries or the O Brien Fleming monitoring boundaries For the remainder of this manual we will refer to them as O Brien Fleming monitoring boundaries Currently the _ AQ User Manual for TSA Copenhagen Trial Unit 2011 O Brien Fleming a spending function is the only a spending function implemented in the TSA software Lis Power family rho 1 gt g F Power family ro Q Brier Fleming p ie r Q v o E E gt Oo aL E 3 OG g o 0 0 0 2 0 4 0 6 0 8 1 0 Information Fraction Figure 6 The shape of the power family a spending functions with p 7 and p 2 and the O Brien Fleming a spending function As shown in figure 6 the O Brien Fleming a spending function is an exponentially increasing function I
149. y setting pop up window for continuous data meta analysis that appears when clicking on the alpha spending button First you will need to give your a spending based test a name e g 5 symmetric O Brien Fleming You will then need to define if you wish to employ a two sided symmetric or one sided test what your overall maximum type error will be what type of a spending you wish to employ currently only the O Brien Fleming function is available You will then need to decide whether you wish to define the information in your meta analysis as the accumulated number of patients sample size accumulated number of events event size or accumulated statistical information Again for one sided tests the Upper one sided test will only test for superiority of the eg ae User Manual for TSA Copenhagen Trial Unit 2011 experimental intervention whereas the Lower will only test for superiority of the control intervention For binary data meta analysis it should be noted that when the outcome is defined as a positive rather than a negative outcome see section 4 1 1 the functions of Upper and Lower are reversed To test for futility i e apply inner wedge futility boundaries check the Apply Inner wedge checkbox The type II error or power for the futility boundaries will automatically be set when you enter your settings for you information size calculation see below Currently the only B spending fun
150. ysis and in the cumulative Z value Trials Ignored x i Following Trials ignored in Interim Looks TSA 1995 Trial 1 2001 Trial 4 Figure 44 Pop up window that inform which interim analyses were removed The data of these trials are however retained in your TSA meta analysis and in the cumulative Z value sple User Manual for TSA Copenhagen Trial Unit 2011 If you have added more than one significance test and do not wish to perform the calculations for all of these you have the option to ignore significance tests To ignore a significance test check the checkbox in the mid column for the row corresponding to the significance test s you wish to ignore Identifier T54 ALPHA ak ee Ipha spending Conventional 2 sided LIL 5 LIL EA A aw of Iterated Logarithm Figure 45 Example of an ignored significance test Conventional 2 sided ignored The cumulative Z curve and the significance boundaries for a spending functions can be displayed using one of three variables on the x axis sample size the event size or the statistical information Significance tests defined on different scales cannot be displayed simultaneously in a graph so you need to select one of these variables for the whole analysis Check the appropriate radio button in the Information Axis area below the Calculation area figure 46 Information axis f Sample size f Event size E Statistical information Figure 46

Download Pdf Manuals

Related Search

Related Contents

Panas。nーc 量・ ョ      Mirage Service Manual  Trilogy/HFN-Trilogy 931  User`s Manual  

Copyright © All rights reserved. Failed to retrieve file