Is it valid to iterate over every permutation of a regression specification and compute an “average...












1












$begingroup$


I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






regression least-squares







share|cite|improve this question









$endgroup$

















    1












    $begingroup$


    I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



    Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



    Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






    regression least-squares







    share|cite|improve this question









    $endgroup$















      1












      1








      1


      1



      $begingroup$


      I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



      Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



      Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






      regression least-squares







      share|cite|improve this question









      $endgroup$




      I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



      Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



      Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






      regression least-squares




      regression least-squares






      share|cite|improve this question













      share|cite|improve this question











      share|cite|improve this question




      share|cite|improve this question










      asked 46 mins ago









      ParseltongueParseltongue

      267114




      267114






















          2 Answers
          2






          active

          oldest

          votes


















          3












          $begingroup$

          One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



          https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



          I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



          https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






          share|cite|improve this answer









          $endgroup$





















            1












            $begingroup$

            Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



            In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






            share|cite|improve this answer









            $endgroup$













              Your Answer





              StackExchange.ifUsing("editor", function () {
              return StackExchange.using("mathjaxEditing", function () {
              StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
              StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
              });
              });
              }, "mathjax-editing");

              StackExchange.ready(function() {
              var channelOptions = {
              tags: "".split(" "),
              id: "65"
              };
              initTagRenderer("".split(" "), "".split(" "), channelOptions);

              StackExchange.using("externalEditor", function() {
              // Have to fire editor after snippets, if snippets enabled
              if (StackExchange.settings.snippets.snippetsEnabled) {
              StackExchange.using("snippets", function() {
              createEditor();
              });
              }
              else {
              createEditor();
              }
              });

              function createEditor() {
              StackExchange.prepareEditor({
              heartbeatType: 'answer',
              autoActivateHeartbeat: false,
              convertImagesToLinks: false,
              noModals: true,
              showLowRepImageUploadWarning: true,
              reputationToPostImages: null,
              bindNavPrevention: true,
              postfix: "",
              imageUploader: {
              brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
              contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
              allowUrls: true
              },
              onDemand: true,
              discardSelector: ".discard-answer"
              ,immediatelyShowMarkdownHelp:true
              });


              }
              });














              draft saved

              draft discarded


















              StackExchange.ready(
              function () {
              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f387998%2fis-it-valid-to-iterate-over-every-permutation-of-a-regression-specification-and%23new-answer', 'question_page');
              }
              );

              Post as a guest















              Required, but never shown

























              2 Answers
              2






              active

              oldest

              votes








              2 Answers
              2






              active

              oldest

              votes









              active

              oldest

              votes






              active

              oldest

              votes









              3












              $begingroup$

              One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



              https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



              I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



              https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






              share|cite|improve this answer









              $endgroup$


















                3












                $begingroup$

                One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



                https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



                I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



                https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






                share|cite|improve this answer









                $endgroup$
















                  3












                  3








                  3





                  $begingroup$

                  One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



                  https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



                  I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



                  https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






                  share|cite|improve this answer









                  $endgroup$



                  One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



                  https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



                  I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



                  https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419







                  share|cite|improve this answer












                  share|cite|improve this answer



                  share|cite|improve this answer










                  answered 27 mins ago









                  eric_kernfelderic_kernfeld

                  2,9821726




                  2,9821726

























                      1












                      $begingroup$

                      Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



                      In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






                      share|cite|improve this answer









                      $endgroup$


















                        1












                        $begingroup$

                        Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



                        In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






                        share|cite|improve this answer









                        $endgroup$
















                          1












                          1








                          1





                          $begingroup$

                          Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



                          In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






                          share|cite|improve this answer









                          $endgroup$



                          Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



                          In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.







                          share|cite|improve this answer












                          share|cite|improve this answer



                          share|cite|improve this answer










                          answered 26 mins ago









                          Bryan KrauseBryan Krause

                          497210




                          497210






























                              draft saved

                              draft discarded




















































                              Thanks for contributing an answer to Cross Validated!


                              • Please be sure to answer the question. Provide details and share your research!

                              But avoid …



                              • Asking for help, clarification, or responding to other answers.

                              • Making statements based on opinion; back them up with references or personal experience.


                              Use MathJax to format equations. MathJax reference.


                              To learn more, see our tips on writing great answers.




                              draft saved


                              draft discarded














                              StackExchange.ready(
                              function () {
                              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f387998%2fis-it-valid-to-iterate-over-every-permutation-of-a-regression-specification-and%23new-answer', 'question_page');
                              }
                              );

                              Post as a guest















                              Required, but never shown





















































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown

































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown







                              -

                              Popular posts from this blog

                              What other Star Trek series did the main TNG cast show up in?

                              Image
                              .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0; } 4 I'm referring to Picard, Riker, Data, Worf, Geordi, Dr. Crusher, Wesley Crusher, Chief O'Brien and family, Deanna Troi, Guinan. I know Picard and Worf show up on DS9, though I haven't seen it myself. I'm curious what happened to the characters after that last episode of TNG? star-trek star-trek-tng share | improve this question edited 10 hours ago Ham Sandwich 6,051 2 25 75
                              Read more

                              Berlina muro

                              Image
                              La berlina muro ĉe Bethaniendamm, 1986 La berlina muro aŭ simple la muro [1] , nomita ankaŭ "kontraŭfaŝisma remparo" en propaganda lingvo de la iama Germana Demokratia Respubliko, estis parto de la interngermana limo, kiu dividis de la 13-a de aŭgusto 1961 ĝis la 9-a de novembro 1989 okcidentan Berlinon de la orienta parto de la urbo kaj de la ĉirkaŭa tereno de la GDR. Ĝi estis unu el la plej famaj simboloj de la malvarma milito kaj de la disdivido de Germanio. Pli ol simpla muro, la berlina muro estis fakte tuta defendokonstruaĵo kun du muroj, ena malplena strio, rondirvojo, gardturoj kaj alarmsistemoj. Dum la provo transiri la severe garditan limon al okcidenta Berlino multaj homoj estis mortigitaj. La preciza nombro de viktimoj estas pridisputita kaj ne certa; la nombroj anoncataj varias inter 86 kaj 238 mortintoj. La malplifortiĝo de Sovetunio kaj la politiko de liberigo gvidita de Miĥail Gorbaĉov ebligis al la orientgermanoj faligi la 9-an de novembro 1989 l
                              Read more

                              Berlina aerponto

                              Image
                              Flugkoridoroj dum la blokado de Berlino La Berlina aerponto estas la provizigo de okcidenta Berlino per aviadiloj fare de la aliancanoj inter la 23-a de junio 1948 kaj la 12-a de majo 1949 dum la blokado de Berlino fare de la sovetia armeo. Enhavo 1 Historio 1.1 La aerkoridoroj 1.2 Unuaj flugoj 1.3 Plibonigo de la aerponto 1.4 Fino de la aerponto 1.5 Rozinbombiloj 1.6 Bilanco 2 Flughavenoj 2.1 Berlino 2.2 Okcidenta Germanio 3 Aviadiloj de la berlina aerponto 4 15-a datreveno de la aerponto 5 Bibliografio 6 Referencoj 7 Eksteraj ligiloj Historio | General Lucius D. Clay, militguberniestro de la usona okupadzono Jam antaŭ la berlina aerponto estiĝis la malgranda aerponto . La estro de la sovetia militadministracio en Germanio, Vasili Daniloviĉ Sokolovski, ordonis la 1-an de aprilo 1948, kiel reago al la konferenco de Londono inter britoj, francoj kaj reprezentantoj de Benelukso pri aligo de Germa
                              Read more