Kdjykuj

I've been reading up on how Mathematica handles contexts, $Context, $ContextPath, and a few of the tutorials they have on Packages.

What I'm wondering about is how the functions defined in, say, CustomPackage` are able to access the variables in CustomPackage`Private`.

For example,

BeginPackage["CustomPackage`"]



MyFunction::usage = "MyFunction[arg1] adds 5 to arg1."



Begin["`Private`"]



abc=5;

MyFunction[arg1_] := arg1 + abc;



End

EndPackage

When I load the package <<CustomPackage` the $ContextPath will have CustomPackage` on it, but not CustomPackage`Private`

So how does MyFunction know the value of abc at the delayed function call (when it is called) if the Private` context isn't on the $ContextPath

So how does MyFunction know the value of abc at the delayed function call (when it is called) if the Private` context isn't on the $ContextPath

There is a misunderstanding here. You are assuming that abc is searched for in some context only when MyFunction[something] is evaluated. This is not the case.

$Context and $ContextPath only affect how source code is parsed (not how expressions are evaluated). In other words, they only affect how the text you write in the package file is interpreted and converted into in-memory expressions. Once the package has been loaded with Get, this interpretation has already happened. MyFunction has been interpreted as the symbol CustomPackage`MyFunction and abc has been interpreted as CustomPackage`Private`abc, according to the value of $Context and $ContextPath at the time each was read. These are the full names of these symbols and this is how they exist in memory.

Load the package and try this:

Block[{$ContextPath},

 Print@Definition[MyFunction]

]

You'll see the following printed:

CustomPackage`MyFunction[CustomPackage`Private`arg1_] := 

  CustomPackage`Private`arg1+CustomPackage`Private`abc

As you can see, a context is always associated with every symbol.

Begin["`Private`"]; sets the current $Context to "CustomPackage `Private`". This causes two things:

• The symbol abc will be searched in the current context first, thus in"CustomPackage`Private`". Only if it is not found there, the search goes on along $ContextPath.




• If no matching symbol is found this way, a new symbol abc is created, namely in the current $Context which is "CustomPackage`Private`". So the full symbol name is "CustomPackage`Private`abc".

For example, running your code in a fresh kernel and executing

??MyFunction

reveals that the full definition of MyFunction is

MyFunction[CustomPackage`Private`arg1_]:=CustomPackage`Private`arg1+CustomPackage`Private`abc

Moreover, with

 ?*`abc

you see that the only symbol in all contexts that matches abc is CustomPackage`Private`abc and has the value 5 assigned to it.

All symbols are created at load time, so when you do:

BeginPackage["X`"];



x::usage="Declaring x as an exported symbol in the X` context";



Begin["`SomePrivateContext`"];



x[a_]:=b



End;



EndPackage;

x was created as X`x but the DownValues of x reference X`SomePrivateContext`a and X`SomePrivateContext`b which were created at the time the function was defined. These symbols are unique, so that reference only ever points that a single object.

So how does MyFunction know the value of abc
at the delayed function call (when it is called)
if the Private` context isn't on the $ContextPath?

because "CustomPackage`Private`" is the value of $Context when MyFunction is defined (i.e. it is not just $ContextPath that determines what a function sees but also what is on $Context).

TL:DR

This is a timely question because it indirectly touches upon the competing imperatives of developers and end-users. To the question itself:

The whole point of packages is that they are a form of encapsulation that allows developers to, without interferance, implement functionality for end-users without bothering them with the underlying details. In particular, the encapsulation involves controlling namespaces so that the underlying details can involve symbols that help implement the functionality but ultimately don't end-up polluting a user's namespace. All symbols defined in a "*`Private`" namespace are created for exactly this purpose.

Hence in the OP's example, the variable abc is an underlying detail for the implementation of the public MyFunction. The developer needs the "detail" of abc but this particular symbol but is of no direct interest to an end-user who typically just ends up calling MyFunction.

The package layout achieves this encapsulation by manipulating $ContextPath and $Context as the control-flow passes through the package when it is first loaded. This is described in the other answers and documentation but it can be useful to see it directly:

loc[n_] := Sow[<|

                 "Location" -> n,

                 "$Context" -> $Context, 

                 "$ContextPath" -> $ContextPath|>];



  Reap[



   loc@1;



   BeginPackage["CustomPackage`"];



   loc@2;



   MyFunction::usage = "MyFunction[arg1] adds 5 to arg1.";



   Begin["`Private`"];



   loc@3;



   abc = 5;

   MyFunction[arg1_] := arg1 + abc;



   End;



   loc@4;



   EndPackage;



   loc@5

   ]// Last // Dataset

When I load the package <the $ContextPath will have CustomPackage on it, but not CustomPackagePrivate

Yes, this implements both the public exporting of all CustomPackage functions but without polluting end-users namespaces with implementation details. In code around Location 3, all packages are cleared out thereby eliminating possible conflicts with existing abc definitions in currently loaded packages. This is encapsulation benefitting developers but the encapsulation benefitting end-users, as observed, is that on exiting (at Location 5) $ContextPath contains "CustomPackage`" (to provide access to MyFunction) but not "CustomPackage`Private`" thereby shielding users from symbols used in MyFunction's implementation.

A programmatic confirmation at Location 5 gives:

{MemberQ["CustomPackage`"]@$ContextPath, 

 MemberQ["CustomPackage`Private`"]@$ContextPath,

 Context["abc"]}



 {True, False, "Global`"}

At Location 3 in the control-flow, the symbol abc is not contained in any of the contexts defined in $ContextPath, ("CustomPackage`", or "System`") nor is it (yet) in the context defined in $Context ("CustomPackage`Private`"). Consequently, the name abc gets created in the context currently set to $Context. At this location $Context has value "CustomPackage`Private`" and hence the symbol CustomPackage`Private`abc is created. When the control flow then moves on to MyFunction, "CustomPackage`Private`" is still the value of $Context so this function "sees" abc (hence it not just $ContextPath that determines what a function sees but what is on both$ContextPath and $Context).

Note how the convention of placing usage definitions at Location 2 is ostensibly for documentation purposes but its more important role is to ensure that the function goes into the package's context (see $Context at Location 2) before subsequently being made available in the implementation and for end-users (see $ContextPath at Locations 3 and 5).

IMO it is kind of cool how these placement protocols just work intuitively without necessarily keeping front-of-mind all the control-flow manipulations, variable-creation mechanisms etc taking place behind the scenes. Hence this means being very careful changing the framework but also IMHO the time is ripe for such extensions given that the line between users/developers may well be in the process of blurring.