What is in your Mathematica tool bag?

We all know that Mathematica is great, but it also often lacks critical functionality. What kind of external packages / tools / resources do you use with Mathematica?

I'll edit (and invite anyone else to do so too) this main post to include resources which are focused on general applicability in scientific research and which as many people as possible will find useful. Feel free to contribute anything, even small code snippets (as I did below for a timing routine).

Also, undocumented and useful features in Mathematica 7 and beyond you found yourself, or dug up from some paper/site are most welcome.

Please include a short description or comment on why something is great or what utility it provides. If you link to books on Amazon with affiliate links please mention it, e.g., by putting your name after the link.

---

Packages:

1. LevelScheme is a package that greatly expands Mathematica's capability to produce good looking plots. I use it if not for anything else then for the much, much improved control over frame/axes ticks. Its newest version is called SciDraw, and it will be released sometime this year.
2. David Park's Presentation Package (US$50 - no charge for updates)
3. Jeremy Michelson's grassmannOps package provides resources for doing algebra and calculus with Grassmann variables and operators that have non trivial commutation relations.
4. John Brown's GrassmannAlgebra package and book for working with Grassmann and Clifford algebras.
5. RISC (Research Institute for Symbolic Computation) has a variety of packages for Mathematica (and other languages) available for download. In particular, there is Theorema for automated theorem proving, and the multitude of packages for symbolic summation, difference equations, etc. at the Algorithmic Combinatorics group's software page.

Tools:

1. MASH is Daniel Reeves's excellent Perl script essentially providing scripting support for Mathematica v7. (Now built in as of Mathematica 8 with the -script option.)
2. An alternate Mathematica shell with a GNU readline input (using python, *nix only)
3. ColourMaths package allows you to visually select parts of an expression and manipulate them. http://www.dbaileyconsultancy.co.uk/colour_maths/colour_maths.html

Resources:

1. Wolfram's own repository MathSource has a lot of useful if narrow notebooks for various applications. Also check out the other sections such as

• Current Documentation,
• Courseware for lectures,
• and Demos for, well, demos.

2. The Mathematica Wikibook.

Books:

1. Mathematica programming: an advanced introduction by Leonid Shifrin (web, pdf) is a must read if you want to do anything more than For loops in Mathematica. We have the pleasure of having Leonid himself answering questions here.
2. Quantum Methods with Mathematica by James F. Feagin (amazon)
3. The Mathematica Book by Stephen Wolfram (amazon) (web)
4. Schaum's Outline (amazon)
5. Mathematica in Action by Stan Wagon (amazon) - 600 pages of neat examples and goes up to Mathematica version 7. Visualization techniques are especially good, you can see some of them on the author's Demonstrations Page.
6. Mathematica Programming Fundamentals by Richard Gaylord (pdf) - A good concise introduction to most of what you need to know about Mathematica programming.
7. Mathematica Cookbook by Sal Mangano published by O'Reilly 2010 832 pages. - Written in the well known O'Reilly Cookbook style: Problem - Solution. For intermediates.
8. Differential Equations with Mathematica, 3rd Ed. Elsevier 2004 Amsterdam by Martha L. Abell, James P. Braselton - 893 pages For beginners, learn solving DEs and Mathematica at the same time.

Undocumented (or scarcely documented) features:

7. How to customize Mathematica keyboard shortcuts. See this question.
8. How to inspect patterns and functions used by Mathematica's own functions. See this answer
9. How to achieve consistent size for GraphPlots in Mathematica? See this question.
10. How to produce documents and presentations with Mathematica. See this question.

One of the nice things about the Mathematica notebook interface is that it can evaluate expressions in any language, not just Mathematica. As a simple example, consider creating a new Shell input cell type that passes the contained expression to the operating system shell for evaluation.

First, define a function that delegates evaluation of a textual command to the external shell:

shellEvaluate[cmd_, _] := Import["!"~~cmd, "Text"]

The second argument is needed and ignored for reasons that will become apparent later. Next, we want to create a new style called Shell:

1. Open a new notebook.
2. Select the menu item Format/Edit Stylesheet...
3. In the dialog, beside Enter a style name: type Shell.
4. Select the cell bracket beside the new style.
5. Select the menu item Cell/Show Expression
6. Overwrite the cell expression with the Step 6 Text given below.
7. Once again, select the menu item Cell/Show Expression
8. Close the dialog.

Use the following cell expression as the Step 6 Text:

Cell[StyleData["Shell"],
 CellFrame->{{0, 0}, {0.5, 0.5}},
 CellMargins->{{66, 4}, {0, 8}},
 Evaluatable->True,
 StripStyleOnPaste->True,
 CellEvaluationFunction->shellEvaluate,
 CellFrameLabels->{{None, "Shell"}, {None, None}},
 Hyphenation->False,
 AutoQuoteCharacters->{},
 PasteAutoQuoteCharacters->{},
 LanguageCategory->"Formula",
 ScriptLevel->1,
 MenuSortingValue->1800,
 FontFamily->"Courier"]

Most of this expression was copied directly form the built-in Program style. The key changes are these lines:

 Evaluatable->True,
 CellEvaluationFunction->shellEvaluate,
 CellFrameLabels->{{None, "Shell"}, {None, None}},

Evaluatable enables the SHIFT+ENTER functionality for the cell. Evaluation will call the CellEvaluationFunction passing the cell content and content type as arguments (shellEvaluate ignores the latter argument). CellFrameLabels is just a nicety that let's the user identify that this cell is unusual.

With all of this in place, we can now enter and evaluate a shell expression:

1. In the notebook created in the steps above, create an empty cell and select the cell bracket.
2. Select the menu item Format/Style/Shell.
3. Type a valid operating system shell command into the cell (e.g. 'ls' on Unix or 'dir' on Windows).
4. Press SHIFT+ENTER.

It is best to keep this defined style in a centrally located stylesheet. Furthermore, evaluation functions like shellEvaluate are best defined as stubs using DeclarePackage in init.m. The details of both of these activities are beyond the scope of this response.

With this functionality, one can create notebooks that contain input expressions in any syntax of interest. The evaluation function can be written in pure Mathematica, or delegate any or all parts of the evaluation to an external agency. Be aware that there are other hooks that relate to cell evaluation, like CellEpilog, CellProlog and CellDynamicExpression.

A common pattern involves writing the input expression text to a temporary file, compiling the file in some language, running the program and capturing the output for ultimate display in the output cell. There are plenty of details to address when implementing a full solution of this kind (like capturing error messages properly), but one must appreciate the fact that it is not only possible to do things like this, but practical.

On a personal note, it is features like this that makes the notebook interface the center of my programming universe.

Update

The following helper function is useful for creating such cells:

evaluatableCell[label_String, evaluationFunction_] :=
  ( CellPrint[
      TextCell[
        ""
      , "Program"
      , Evaluatable -> True
      , CellEvaluationFunction -> (evaluationFunction[#]&)
      , CellFrameLabels -> {{None, label}, {None, None}}
      , CellGroupingRules -> "InputGrouping"
      ]
    ]
  ; SelectionMove[EvaluationNotebook[], All, EvaluationCell]
  ; NotebookDelete[]
  ; SelectionMove[EvaluationNotebook[], Next, CellContents]
  )

It is used thus:

shellCell[] := evaluatableCell["shell", Import["!"~~#, "Text"] &]

Now, if shellCell[] is evaluated, the input cell will be deleted and replaced with a new input cell that evaluates its contents as a shell command.

Todd Gayley (Wolfram Research) just send me a nice hack which allows to "wrap" built-in functions with arbitrary code. I feel that I have to share this useful instrument. The following is Todd's answer on my question.

>A bit of interesting (?) history: That style of hack for "wrapping" a >built-in function was invented around 1994 by Robby Villegas and I, >ironically for the function Message, in a package called ErrorHelp >that I wrote for the Mathematica Journal back then. It has been used >many times, by many people, since then. It's a bit of an insider's >trick, but I think it's fair to say that it has become the canonical >way of injecting your own code into the definition of a built-in >function. It gets the job done nicely. You can, of course, put the >$inMsg variable into any private context you wish.

Unprotect[Message];

Message[args___] := Block[{$inMsg = True, result},
   "some code here";
   result = Message[args];
   "some code here";
   result] /; ! TrueQ[$inMsg]

Protect[Message];

I've mentioned this before, but the tool I find most useful is an application of Reap and Sow which mimics/extends the behavior of GatherBy:

SelectEquivalents[x_List,f_:Identity, g_:Identity, h_:(#2&)]:=
   Reap[Sow[g[#],{f[#]}]&/@x, _, h][[2]];

This allows me to group lists by any criteria and transform them in the process. The way it works is that a criteria function (f) tags each item in the list, each item is then transformed by a second supplied function (g), and the specific output is controlled by a third function (h). The function h accepts two arguments: a tag and a list of the collected items that have that tag. The items retain their original order, so if you set h = #1& then you get an unsorted Union, like in the examples for Reap. But, it can be used for secondary processing.

As an example of its utility, I've been working with Wannier90 which outputs the spatially dependent Hamiltonian into a file where each line is a different element in the matrix, as follows

rx ry rz i j Re[Hij] Im[Hij]

To turn that list into a set of matrices, I gathered up all sublists that contain the same coordinate, turned the element information into a rule (i.e. {i,j}-> Re[Hij]+I Im[Hij]), and then turned the collected rules into a SparseArray all with the one liner:

SelectEquivalents[hamlst, 
      #[[;; 3]] &, 
      #[[{4, 5}]] -> (Complex @@ #[[6 ;;]]) &, 
      {#1, SparseArray[#2]} &]

Honestly, this is my Swiss Army Knife, and it makes complex things very simple. Most of my other tools are somewhat domain specific, so I'll probably not post them. However, most, if not all, of them reference SelectEquivalents.

Edit: it doesn't completely mimic GatherBy in that it cannot group multiple levels of the expression as simply as GatherBy can. However, Map works just fine for most of what I need.

Example: @Yaroslav Bulatov has asked for a self-contained example. Here's one from my research that has been greatly simplified. So, let's say we have a set of points in a plane

In[1] := pts = {{-1, -1, 0}, {-1, 0, 0}, {-1, 1, 0}, {0, -1, 0}, {0, 0, 0}, 
 {0, 1, 0}, {1, -1, 0}, {1, 0, 0}, {1, 1, 0}}

and we'd like to reduce the number of points by a set of symmetry operations. (For the curious, we are generating the little group of each point.) For this example, let's use a four fold rotation axis about the z-axis

In[2] := rots = RotationTransform[#, {0, 0, 1}] & /@ (Pi/2 Range[0, 3]);

Using SelectEquivalents we can group the points that produce the same set of images under these operations, i.e. they're equivalent, using the following

In[3] := SelectEquivalents[ pts, Union[Through[rots[#] ] ]& ] (*<-- Note Union*)
Out[3]:= {{{-1, -1, 0}, {-1, 1, 0}, {1, -1, 0}, {1, 1, 0}},
          {{-1, 0, 0}, {0, -1, 0}, {0, 1, 0}, {1, 0, 0}},
          {{0,0,0}}}

which produces 3 sublists containing the equivalent points. (Note, Union is absolutely vital here as it ensures that the same image is produced by each point. Originally, I used Sort, but if a point lies on a symmetry axis, it is invariant under the rotation about that axis giving an extra image of itself. So, Union eliminates these extra images. Also, GatherBy would produce the same result.) In this case, the points are already in a form that I will use, but I only need a representative point from each grouping and I'd like a count of the equivalent points. Since, I don't need to transform each point, I use the Identity function in the second position. For the third function, we need to be careful. The first argument passed to it will be the images of the points under the rotations which for the point {0,0,0} is a list of four identical elements, and using it would throw off the count. However, the second argument is just a list of all the elements that have that tag, so it will only contain {0,0,0}. In code,

In[4] := SelectEquivalents[pts,  
             Union[Through[rots[#]]]&, #&, {#2[[1]], Length[#2]}& ]
Out[4]:= {{{-1, -1, 0}, 4}, {{-1, 0, 0}, 4}, {{0, 0, 0}, 1}}

Note, this last step can just as easily be accomplished by

In[5] := {#[[1]], Length[#]}& /@ Out[3]

But, it is easy with this and the less complete example above to see how very complex transformations are possible with a minimum of code.

This is not a complete resource, so I'm throwing it here in the answers section, but I have found it very useful when figuring out speed issues (which, unfortunately, is a large part of what Mathematica programming is about).

timeAvg[func_] := Module[
{x = 0, y = 0, timeLimit = 0.1, p, q, iterTimes = Power[10, Range[0, 10]]},
Catch[ If[(x = First[Timing[(y++; Do[func, {#}]);]]) > timeLimit,
    Throw[{x, y}]
    ] & /@ iterTimes
 ] /. {p_, q_} :> p/iterTimes[[q]]
];
Attributes[timeAvg] = {HoldAll};

Usage is then simply timeAvg@funcYouWantToTest.

EDIT: Mr. Wizard has provided a simpler version that does away with Throw and Catch and is a bit easier to parse:

SetAttributes[timeAvg, HoldFirst]
timeAvg[func_] := Do[If[# > 0.3, Return[#/5^i]] & @@ 
                     Timing @ Do[func, {5^i}]
                     ,{i, 0, 15}]

EDIT: Here's a version from acl (taken from here):

timeIt::usage = "timeIt[expr] gives the time taken to execute expr, \
  repeating as many times as necessary to achieve a total time of 1s";

SetAttributes[timeIt, HoldAll]
timeIt[expr_] := Module[{t = Timing[expr;][[1]], tries = 1},
  While[t < 1., tries *= 2; t = Timing[Do[expr, {tries}];][[1]];]; 
  t/tries]

Internal`InheritedBlock

I have learned recently the existence of such useful function as Internal`InheritedBlock, from this message of Daniel Lichtblau in the official newsgroup.

As I understand, Internal`InheritedBlock allows to pass a copy of an outbound function inside the Block scope:

In[1]:= Internal`InheritedBlock[{Message},
Print[Attributes[Message]];
Unprotect[Message];
Message[x___]:=Print[{{x},Stack[]}];
Sin[1,1]
]
Sin[1,1]
During evaluation of In[1]:= {HoldFirst,Protected}
During evaluation of In[1]:= {{Sin::argx,Sin,2},{Internal`InheritedBlock,CompoundExpression,Sin,Print,List}}
Out[1]= Sin[1,1]
During evaluation of In[1]:= Sin::argx: Sin called with 2 arguments; 1 argument is expected. >>
Out[2]= Sin[1,1]

I think this function can be very useful for everyone who need to modify built-in functions temporarily!

Comparison with Block

Let us define some function:

a := Print[b]

Now we wish to pass a copy of this function into the Block scope. The naive trial does not give what we want:

In[2]:= Block[{a = a}, OwnValues[a]]

During evaluation of In[9]:= b

Out[2]= {HoldPattern[a] :> Null}

Now trying to use delayed definition in the first argument of Block (it is an undocumented feature too):

In[3]:= Block[{a := a}, OwnValues[a]]
Block[{a := a}, a]

Out[3]= {HoldPattern[a] :> a}

During evaluation of In[3]:= b

We see that in this case a works but we have not got a copy of the original a inside of the Block scope.

Now let us try Internal`InheritedBlock:

In[5]:= Internal`InheritedBlock[{a}, OwnValues[a]]

Out[5]= {HoldPattern[a] :> Print[b]}

We have got a copy of the original definition for a inside of the Block scope and we may modify it in the way we want without affecting the global definition for a!

Mathematica is a sharp tool, but it can cut you with its somewhat untyped behaviour and avalanches of cryptic diagnostic messages. One way to deal with this is to define functions following this idiom:

ClearAll@zot
SetAttributes[zot, ...]
zot[a_] := ...
zot[b_ /; ...] := ...
zot[___] := (Message[zot::invalidArguments]; Abort[])

That is a lot of boilerplate, which I'm frequently tempted to skip. Especially when prototyping, which happens a lot in Mathematica. So, I use a macro called define that allows me to stay disciplined, with much less boilerplate.

A basic usage of define is like this:

define[  fact[0] = 1
; fact[n_ /; n > 0] := n * fact[n-1]
]

fact[5]

120

It doesn't look like much at first, but there are some hidden benefits. The first service that define provides is that it automatically applies ClearAll to the symbol being defined. This ensures that there are no leftover definitions -- a common occurrence during the initial development of a function.

The second service is that the function being defined is automatically "closed". By this I mean that the function will issue a message and abort if it is invoked with an argument list that is not matched by one of the definitions:

fact[-1]

define::badargs: There is no definition for 'fact' applicable to fact[-1].
$Aborted

This is the primary value of define, which catches a very common class of error.

Another convenience is a concise way to specify attributes on the function being defined. Let's make the function Listable:

define[  fact[0] = 1
; fact[n_ /; n > 0] := n * fact[n-1]
, Listable
]

fact[{3, 5, 8}]

{6, 120, 40320}

In addition to all of the normal attributes, define accepts an additional attribute called Open. This prevents define from adding the catch-all error definition to the function:

define[  successor[x_ /; x > 0] := x + 1
, Open
]

successor /@ {1, "hi"}

{2, successor["hi"]}

Multiple attributes may be defined for a function:

define[  flatHold[x___] := Hold[x]
, {Flat, HoldAll}
]

flatHold[flatHold[1+1, flatHold[2+3]], 4+5]

Hold[1 + 1, 2 + 3, 4 + 5]

Without further ado, here is the definition of define:

ClearAll@define
SetAttributes[define, HoldAll]
define[body_, attribute_Symbol] := define[body, {attribute}]
define[body:(_Set|_SetDelayed), attributes_List:{}] := define[CompoundExpression[body], attributes]
define[body:CompoundExpression[((Set|SetDelayed)[name_Symbol[___], _])..], attributes_List:{}] :=
  ( ClearAll@name
  ; SetAttributes[name, DeleteCases[attributes, Open]]
  ; If[!MemberQ[attributes, Open]
    , def:name[___] := (Message[define::badargs, name, Defer@def]; Abort[])
    ]
  ; body
  ;
  )
def:define[___] := (Message[define::malformed, Defer@def]; Abort[])

define::badargs = "There is no definition for '``' applicable to ``.";
define::malformed = "Malformed definition: ``";

The exhibited implementation supports neither up-values nor currying, nor patterns more general than simple function definition. It remains useful, however.

Start without a blank notebook open

I was bothered by having Mathematica start with a blank notebook open. I could close this notebook with a script, but it would still flash open briefly. My hack is to create a file Invisible.nb containing:

Notebook[{},Visible->False]

And add this to my Kernel\init.m :

If[Length[Notebooks["Invisible*"]] > 0, 
  NotebookClose[Notebooks["Invisible*"][[1]]]
]

SetOptions[$FrontEnd,  Options[$FrontEnd, NotebooksMenu] /. 
    HoldPattern["Invisible.nb" -> {__}] :> Sequence[]
]

I now start Mathematica by opening Invisible.nb

There may be a better way, but this has served me well.

---

Customized Fold and FoldList

Fold[f, x] is made equivalent to Fold[f, First@x, Rest@x]

Incidentally, I believe this may find its way into a future version of Mathematica.

Surprise! This has been implemented, though it is presently undocumented. I am informed that it was implemented in 2011 by Oliver Ruebenkoenig, apparently not long after I posted this. Thank you Oliver Ruebenkoenig!

Unprotect[Fold, FoldList]

Fold[f_, h_[a_, b__]] := Fold[f, Unevaluated @ a, h @ b]
FoldList[f_, h_[a_, b__]] := FoldList[f, Unevaluated @ a, h @ b]

(* Faysal's recommendation to modify SyntaxInformation *)
SyntaxInformation[Fold]     = {"ArgumentsPattern" -> {_, _, _.}};
SyntaxInformation[FoldList] = {"ArgumentsPattern" -> {_, _., {__}}};

Protect[Fold, FoldList]

Updated to allow this:

SetAttributes[f, HoldAll]
Fold[f, Hold[1 + 1, 2/2, 3^3]]

> f[f[1 + 1, 2/2], 3^3]

---

"Dynamic Partition"

See Mathematica.SE post #7512 for a new version of this function.

Frequently I want to partition a list according to a sequence of lengths.

pseudo-code example:

partition[{1,2,3,4,5,6}, {2,3,1}]

Output: {{1,2}, {3,4,5}, {6}}

I came up with this:

dynP[l_, p_] := 
 MapThread[l[[# ;; #2]] &, {{0} ~Join~ Most@# + 1, #} &@Accumulate@p]

Which I then completed with this, including argument testing:

dynamicPartition[l_List, p : {_Integer?NonNegative ..}] :=
  dynP[l, p] /; Length@l >= Tr@p

dynamicPartition[l_List, p : {_Integer?NonNegative ..}, All] :=
  dynP[l, p] ~Append~ Drop[l, Tr@p] /; Length@l >= Tr@p

dynamicPartition[l_List, p : {_Integer?NonNegative ..}, n__ | {n__}] :=
  dynP[l, p] ~Join~ Partition[l ~Drop~ Tr@p, n] /; Length@l >= Tr@p

The third argument controls what happens to elements beyond the split specification.

---

Szabolcs's Mathematica tricks

The one I use most frequently is the Paste Tabular Data Palette

CreatePalette@
 Column@{Button["TSV",     Module[{data, strip},      data = NotebookGet[ClipboardNotebook[]][[1, 1, 1]];
     strip[s_String] := 
      StringReplace[s, RegularExpression["^\\s*(.*?)\\s*$"] -> "$1"];
     strip[e_] := e;
     If[Head[data] === String, 
      NotebookWrite[InputNotebook[], 
       ToBoxes@Map[strip, ImportString[data, "TSV"], {2}]]]]], 
   Button["CSV",     Module[{data, strip},      data = NotebookGet[ClipboardNotebook[]][[1, 1, 1]];
     strip[s_String] := 
      StringReplace[s, RegularExpression["^\\s*(.*?)\\s*$"] -> "$1"];
     strip[e_] := e;
     If[Head[data] === String, 
      NotebookWrite[InputNotebook[], 
       ToBoxes@Map[strip, ImportString[data, "CSV"], {2}]]]]], 
   Button["Table",     Module[{data}, data = NotebookGet[ClipboardNotebook[]][[1, 1, 1]];
     If[Head[data] === String, 
      NotebookWrite[InputNotebook[], 
       ToBoxes@ImportString[data, "Table"]]]]]}

---

Modify external data from within Compile

Recently Daniel Lichtblau showed this method I had never seen before. In my opinion it significantly extends the utility of Compile

ll = {2., 3., 4.};
c = Compile[{{x}, {y}}, ll[[1]] = x; y];

c[4.5, 5.6]

ll

(* Out[1] = 5.6  *)

(* Out[2] = {4.5, 3., 4.}  *)

General PDF/EMF export problems and solutions

1. It is completely unexpected and undocumented, but Mathematica exports and saves graphics in PDF and EPS formats using a set of style definitions that differs from the one used for displaying Notebooks on screen. By default Notebooks are displayed on screen in the "Working" style environment (which is default value for the ScreenStyleEvironment global $FrontEnd option) but are printed in the "Printout" style environment (which is default value for the PrintingStyleEnvironment global $FrontEnd option). When one exports graphics in raster formats such as GIF and PNG or in EMF format Mathematica generates graphics that looks exactly like it looks inside Notebook. It seems that the "Working" style environment is used for rendering in this case. But it is not the case when you export/save anything in PDF or EPS formats! In this case the "Printout" style environment is used by default that differs very deeply from the "Working" style environment. First of all, the "Printout" style environment sets Magnification to 80%. Secondly, it uses its own values for the font sizes of different styles and this results in inconsistent font size changes in the genarated PDF file as compared with the original on-screen representation. The latter can be called FontSize fluctuations which are very annoying. But happily this can be avoided by setting the PrintingStyleEnvironment global $FrontEnd option to "Working":

   SetOptions[$FrontEnd, PrintingStyleEnvironment -> "Working"]

2. The common problem with exporting to EMF format is that most of programs (not only Mathematica) generate a file that looks nice at the default size but becomes ugly when you zoom it in. It is because metafiles are sampled at screen resolution fidelity. The quality of the generated EMF file can be enhanced by Magnifying the original graphical object so that exactness of sampling of the original graphics becomes much more precise. Compare two files:

   graphics1 = First@ImportString[ ExportString[Style["a", FontFamily -> "Times"], "PDF"], "PDF"]; graphics2 = Magnify[graphics1, 10]; Export["C:\test1.emf", graphics1] Export["C:\test2.emf", graphics2]

If you insert these files into Microsoft Word and zoom them in you will see that the first "a" has sawtooth on it while the second has not (tested with Mathematica 6).

Another way through ImageResolution was suggested by Chris Degnen (this option has effect at least starting from Mathematica 8):

Export["C:\\test1.emf", graphics1]
Export["C:\\test2.emf", graphics1, ImageResolution -> 300]

3) In Mathematica we have three ways to convert graphics into metafile: via Export to "EMF" (strongly recommended way: produces metafile with highest possible quality), via Save selection As... menu item (produces much lesser precise figure, not recommended) and via Edit ► Copy As ► Metafile menu item (I strongly recommend against this route).

By popular demand, the code to generate the top-10 SO answerers plot (except annotations) using the SO API.

getRepChanges[userID_Integer] :=
 Module[{totalChanges},
  totalChanges = 
   "total" /. 
    Import["http://api.stackoverflow.com/1.1/users/" <> 
      ToString[userID] <> "/reputation?fromdate=0&pagesize=10&page=1",
      "JSON"];
  Join @@ Table[
    "rep_changes" /. 
     Import["http://api.stackoverflow.com/1.1/users/" <> 
       ToString[userID] <> 
       "/reputation?fromdate=0&pagesize=10&page=" <> ToString[page], 
      "JSON"],
    {page, 1, Ceiling[totalChanges/10]}
    ]
  ]

topAnswerers = ({"display_name", 
      "user_id"} /. #) & /@ ("user" /. ("top_users" /. 
      Import["http://api.stackoverflow.com/1.1/tags/mathematica/top-\
answerers/all-time", "JSON"]))

repChangesTopUsers =
  Monitor[Table[
    repChange = 
     ReleaseHold[(Hold[{DateList[
              "on_date" + AbsoluteTime["January 1, 1970"]], 
             "positive_rep" - "negative_rep"}] /. #) & /@ 
        getRepChanges[userID]] // Sort;
    accRepChange = {repChange[[All, 1]], 
       Accumulate[repChange[[All, 2]]]}\[Transpose],
    {userID, topAnswerers[[All, 2]]}
    ], userID];

pl = DateListLogPlot[
  Tooltip @@@ 
   Take[({repChangesTopUsers, topAnswerers[[All, 1]]}\[Transpose]), 
    10], Joined -> True, Mesh -> None, ImageSize -> 1000, 
  PlotRange -> {All, {10, All}}, 
  BaseStyle -> {FontFamily -> "Arial-Bold", FontSize -> 16}, 
  DateTicksFormat -> {"MonthNameShort", " ", "Year"}, 
  GridLines -> {True, None}, 
  FrameLabel -> (Style[#, FontSize -> 18] & /@ {"Date", "Reputation", 
      "Top-10 answerers", ""})]

Caching expressions

I find these functions very helpful to cache any expression. The interesting thing here for these two functions is that the held expression itself is used as a key of the hashtable/symbol Cache or CacheIndex, compared to the well-known memoization in mathematica where you can only cache result if the function is defined like f[x_] := f[x] = ... So you can cache any part of a code, this is useful if a function is to be called several times but just some parts of the code must not be recomputed.

To cache an expression independently of its arguments.

SetAttributes[Cache, HoldFirst];
c:Cache[expr_] := c = expr;

Ex: Cache[Pause[5]; 6]
Cache[Pause[5]; 6]

The second time the expression returns 6 without waiting.

To cache an expression using an alias expression that can depend on an argument of the cached expression.

SetAttributes[CacheIndex, HoldRest];
c:CacheIndex[index_,expr_] := c = expr;

Ex: CacheIndex[{"f",2},x=2;y=4;x+y]

If expr takes some time to compute, it is much faster to evaluate {"f",2} for example to retrieve the cached result.

For a variation of these functions in order to have a localized cache (ie. the cache memory is automatically released outside the Block construct) see this post https://stackoverflow.com/questions/7781981/avoid-repreated-calls-to-interpolation/7784989#7784989

Deleting cached values

To delete cached values when you don't know the number of definitions of a function. I consider that definitions have a Blank somewhere in their arguments.

DeleteCachedValues[f_] := 
       DownValues[f] = Select[DownValues[f], !FreeQ[Hold@#,Pattern]&];

To delete cached values when you know the number of definitions of a function (goes slightly faster).

DeleteCachedValues[f_,nrules_] := 
       DownValues[f] = Extract[DownValues[f], List /@ Range[-nrules, -1]];

This uses the fact that definitions of a function are at the end of their DownValues list, cached values are before.

Using symbols to store data and object-like functions

Also here are interesting functions to use symbols like objects.

It is already well known that you can store data in symbols and quickly access them using DownValues

mysymbol["property"]=2;

You can access the list of keys (or properties) of a symbol using these functions based on what dreeves submitted in a post on this site:

SetAttributes[RemoveHead, {HoldAll}];
RemoveHead[h_[args___]] := {args};
NKeys[symbol_] := RemoveHead @@@ DownValues[symbol(*,Sort->False*)][[All,1]];
Keys[symbol_] := NKeys[symbol] /. {x_} :> x;

I use this function a lot to display all infos contained in the DownValues of a symbol:

PrintSymbol[symbol_] :=
  Module[{symbolKeys},
    symbolKeys = Keys[symbol];
    TableForm@Transpose[{symbolKeys, symbol /@ symbolKeys}]
  ];

Finally here is a simple way to create a symbol that behaves like an object in object oriented programming (it just reproduces the most basic behaviour of OOP but I find the syntax elegant) :

Options[NewObject]={y->2};
NewObject[OptionsPattern[]]:=
  Module[{newObject},
    newObject["y"]=OptionValue[y];

    function[newObject,x_] ^:= newObject["y"]+x;
    newObject /: newObject.function2[x_] := 2 newObject["y"]+x;

    newObject
  ];

Properties are stored as DownValues and methods as delayed Upvalues in the symbol created by Module that is returned. I found the syntax for function2 that is the usual OO-syntax for functions in https://stackoverflow.com/questions/6097071/tree-data-structure-in-mathematica/6097444#6097444.

For a list of existing types of values each symbol has, see http://reference.wolfram.com/mathematica/tutorial/PatternsAndTransformationRules.html and http://www.verbeia.com/mathematica/tips/HTMLLinks/Tricks_Misc_4.html.

For example try this

x = NewObject[y -> 3];
function[x, 4]
x.function2[5]

You can go further if you want to emulate object inheritance using a package called InheritRules available here http://library.wolfram.com/infocenter/MathSource/671/

You could also store the function definition not in newObject but in a type symbol, so if NewObject returned type[newObject] instead of newObject you could define function and function2 like this outside of NewObject (and not inside) and have the same usage as before.

function[type[object_], x_] ^:= object["y"] + x;
type /: type[object_].function2[x_] := 2 object["y"]+x;

Use UpValues[type] to see that function and function2 are defined in the type symbol.

Further ideas about this last syntax are introduced here https://mathematica.stackexchange.com/a/999/66.

Improved version of SelectEquivalents

@rcollyer: Many thanks for bringing SelectEquivalents to the surface, it's an amazing function. Here is an improved version of SelectEquivalents listed above with more possibilities and using options, this makes it easier to use.

Options[SelectEquivalents] = 
   {
      TagElement->Identity,
      TransformElement->Identity,
      TransformResults->(#2&) (*#1=tag,#2 list of elements corresponding to tag*),
      MapLevel->1,
      TagPattern->_,
      FinalFunction->Identity
   };

SelectEquivalents[x_List,OptionsPattern[]] := 
   With[
      {
         tagElement=OptionValue@TagElement,
         transformElement=OptionValue@TransformElement,
         transformResults=OptionValue@TransformResults,
         mapLevel=OptionValue@MapLevel,
         tagPattern=OptionValue@TagPattern,
         finalFunction=OptionValue@FinalFunction
      }
      ,
      finalFunction[
         Reap[
            Map[
               Sow[
                  transformElement@#
                  ,
                  {tagElement@#}
               ]&
               , 
               x
               , 
               {mapLevel}
            ] 
            , 
            tagPattern
            , 
            transformResults
         ][[2]]
      ]
   ];

Here are examples of how this version can be used:

https://stackoverflow.com/questions/6974544/using-mathematica-gather-collect-properly/6987982#6987982

https://stackoverflow.com/questions/8099866/how-would-you-do-a-pivottable-function-in-mathematica/8105565#8105565

https://stackoverflow.com/questions/8178714/mathematica-fast-2d-binning-algorithm/8183307#8183307

Internal`Bag

Daniel Lichtblau describes here an interesting internal data structure for growing lists.

https://stackoverflow.com/questions/6691491/implementing-a-quadtree-in-mathematica/6795762#6795762

Debugging functions

These two posts point to useful functions for debugging:

https://stackoverflow.com/questions/6167291/how-to-debug-when-writting-small-or-big-codes-using-mathematica-workbench-mma/6168036#6168036 (ShowIt)

https://stackoverflow.com/questions/5459735/the-clearest-way-to-represent-mathematicas-evaluation-sequence/5527117#5527117 (TraceView)

Here's another function based on Reap and Sow to extract expressions from different parts of a program and store them in a symbol.

SetAttributes[ReapTags,HoldFirst];
ReapTags[expr_]:=
   Module[{elements},
      Reap[expr,_,(elements[#1]=#2/.{x_}:>x)&];
      elements
   ];

Here's an example

ftest[]:=((*some code*)Sow[1,"x"];(*some code*)Sow[2,"x"];(*some code*)Sow[3,"y"]);
s=ReapTags[ftest[]];
Keys[s]
s["x"]
PrintSymbol[s] (*Keys and PrintSymbol are defined above*)

Other resources

Here's a list of interesting links for learning purpose:

A collection of Mathematica learning resources

Updated here: https://mathematica.stackexchange.com/a/259/66

My utility functions (I have these built in to MASH, which is mentioned in the question):

pr = WriteString["stdout", ##]&;            (* More                           *)
prn = pr[##, "\n"]&;                        (*  convenient                    *)
perr = WriteString["stderr", ##]&;          (*   print                        *)
perrn = perr[##, "\n"]&;                    (*    statements.                 *)
re = RegularExpression;                     (* I wish mathematica             *)
eval = ToExpression[cat[##]]&;              (*  weren't so damn               *)
EOF = EndOfFile;                            (*   verbose!                     *)
read[] := InputString[""];                  (* Grab a line from stdin.        *)
doList[f_, test_] :=                        (* Accumulate list of what f[]    *)
  Most@NestWhileList[f[]&, f[], test];      (*  returns while test is true.   *)
readList[] := doList[read, #=!=EOF&];       (* Slurp list'o'lines from stdin. *)
cat = StringJoin@@(ToString/@{##})&;        (* Like sprintf/strout in C/C++.  *)
system = Run@cat@##&;                       (* System call.                   *)
backtick = Import[cat["!", ##], "Text"]&;   (* System call; returns stdout.   *)
slurp = Import[#, "Text"]&;                 (* Fetch contents of file as str. *)
                                            (* ABOVE: mma-scripting related.  *)
keys[f_, i_:1] :=                           (* BELOW: general utilities.      *)
  DownValues[f, Sort->False][[All,1,1,i]];  (* Keys of a hash/dictionary.     *)
SetAttributes[each, HoldAll];               (* each[pattern, list, body]      *)
each[pat_, lst_, bod_] := ReleaseHold[      (*  converts pattern to body for  *)
  Hold[Cases[Evaluate@lst, pat:>bod];]];    (*   each element of list.        *)
some[f_, l_List] := True ===                (* Whether f applied to some      *)
  Scan[If[f[#], Return[True]]&, l];         (*  element of list is True.      *)
every[f_, l_List] := Null ===               (* Similarly, And @@ f/@l         *)
  Scan[If[!f[#], Return[False]]&, l];       (*  (but with lazy evaluation).   *)

One trick I've used, which allows you to emulate the way most built-in functions work with bad arguments (by sending a message and then returning the whole form unevaluated) exploits a quirk of the way Condition works when used in a defintion. If foo should only work with one argument:

foo[x_] := x + 1;
expr : foo[___] /; (Message[foo::argx, foo, Length@Unevaluated[expr], 1]; 
                    False) := Null; (* never reached *)

If you have more complex needs, it's easy to factor out the argument validation and message generation as an independent function. You can do more elaborate things by using side effects in Condition beyond just generating messages, but in my opinion most of them fall into the "sleazy hack" category and should be avoided if possible.

Also, in the "metaprogramming" category, if you have a Mathematica package (.m) file, you can use the "HeldExpressions" element to get all the expressions in the file wrapped in HoldComplete. This makes tracking things down much easier than using text-based searches. Unfortunately, there's no easy way to do the same thing with a notebook, but you can get all the input expressions using something like the following:

inputExpressionsFromNotebookFile[nb_String] :=
 Cases[Get[nb],
  Cell[BoxData[boxes_], "Input", ___] :>
   MakeExpression[StripBoxes[boxes], StandardForm],
  Infinity]

Lastly, you can use the fact that Module emulates lexical closures to create the equivalent of reference types. Here's a simple stack (which uses a variation the Condition trick for error handling as a bonus):

ClearAll[MakeStack, StackInstance, EmptyQ, Pop, Push, Peek]
 With[{emptyStack = Unique["empty"]},
  Attributes[StackInstance] = HoldFirst;
  MakeStack[] :=
   Module[{backing = emptyStack},
    StackInstance[backing]];

  StackInstance::empty = "stack is empty";

  EmptyQ[StackInstance[backing_]] := (backing === emptyStack);

  HoldPattern[
    Pop[instance : StackInstance[backing_]]] /;
    ! EmptyQ[instance] || (Message[StackInstance::empty]; False) :=
   (backing = Last@backing; instance);

  HoldPattern[Push[instance : StackInstance[backing_], new_]] :=
   (backing = {new, backing}; instance);

  HoldPattern[Peek[instance : StackInstance[backing_]]] /;
    ! EmptyQ[instance] || (Message[StackInstance::empty]; False) :=
   First@backing]

Now you can print the elements of a list in reverse order in a needlessly convoluted way!

With[{stack = MakeStack[], list},
 Do[Push[stack, elt], {elt, list}];

 While[!EmptyQ[stack],
  Print[Peek@stack];
  Pop@stack]]

Printing system symbol definitions without context prepended

The contextFreeDefinition[] function below will attempt to print the definition of a symbol without the most common context prepended. The definition then can be copied to Workbench and formatted for readability (select it, right click, Source -> Format)

Clear[commonestContexts, contextFreeDefinition]

commonestContexts[sym_Symbol, n_: 1] := Quiet[
  Commonest[
   Cases[Level[DownValues[sym], {-1}, HoldComplete], 
    s_Symbol /; FreeQ[$ContextPath, Context[s]] :> Context[s]], n],
  Commonest::dstlms]

contextFreeDefinition::contexts = "Not showing the following contexts: `1`";

contextFreeDefinition[sym_Symbol, contexts_List] := 
 (If[contexts =!= {}, Message[contextFreeDefinition::contexts, contexts]];
  Internal`InheritedBlock[{sym}, ClearAttributes[sym, ReadProtected];
   Block[{$ContextPath = Join[$ContextPath, contexts]}, 
    Print@InputForm[FullDefinition[sym]]]])

contextFreeDefinition[sym_Symbol, context_String] := 
 contextFreeDefinition[sym, {context}]

contextFreeDefinition[sym_Symbol] := 
 contextFreeDefinition[sym, commonestContexts[sym]]

---

withRules[]

Caveat: This function does not localize variables the same way With and Module do, which means that nested localization constructs won't work as expected. withRules[{a -> 1, b -> 2}, With[{a=3}, b_ :> b]] will replace a and b in the nested With and Rule, while With doesn't do this.

This is a variant of With that uses rules instead of = and :=:

ClearAll[withRules]
SetAttributes[withRules, HoldAll]
withRules[rules_, expr_] :=
  Internal`InheritedBlock[
    {Rule, RuleDelayed},
    SetAttributes[{Rule, RuleDelayed}, HoldFirst];
    Unevaluated[expr] /. rules
  ]

I found this useful while cleaning up code written during experimentation and localizing variables. Occasionally I end up with parameter lists in the form of {par1 -> 1.1, par2 -> 2.2}. With withRules parameter values are easy to inject into code previously written using global variables.

Usage is just like With:

withRules[
  {a -> 1, b -> 2},
  a+b
]

---

Antialiasing 3D graphics

This is a very simple technique to antialias 3D graphics even if your graphics hardware doesn't support it natively.

antialias[g_, n_: 3] := 
  ImageResize[Rasterize[g, "Image", ImageResolution -> n 72], Scaled[1/n]]

Here's an example:

Note that a large value for n or a large image size tends to expose graphics driver bugs or introduce artefacts.

---

Notebook diff functionality

Notebook diff functionality is available in the <<AuthorTools` package, and (at least in version 8) in the undocumented NotebookTools` context. This is a little GUI to diff two notebooks that are currently open:

PaletteNotebook@DynamicModule[  {nb1, nb2},   Dynamic@Column[    {PopupMenu[Dynamic[nb1], 
      Thread[Notebooks[] -> NotebookTools`NotebookName /@ Notebooks[]]], 
     PopupMenu[Dynamic[nb2], 
      Thread[Notebooks[] -> NotebookTools`NotebookName /@ Notebooks[]]], 
     Button["Show differences",       CreateDocument@NotebookTools`NotebookDiff[nb1, nb2]]}]
  ]

Recursive pure functions (#0) seem to be one of the darker corners of the language. Here are a couple of non-trivial examples of their use , where this is really useful (not that they can not be done without it). The following is a pretty concise and reasonably fast function to find connected components in a graph, given a list of edges specified as pairs of vertices:

ClearAll[setNew, componentsBFLS];
setNew[x_, x_] := Null;
setNew[lhs_, rhs_]:=lhs:=Function[Null, (#1 := #0[##]); #2, HoldFirst][lhs, rhs];

componentsBFLS[lst_List] := Module[{f}, setNew @@@ Map[f, lst, {2}];
   GatherBy[Tally[Flatten@lst][[All, 1]], f]];

What happens here is that we first map a dummy symbol on each of the vertex numbers, and then set up a way that, given a pair of vertices {f[5],f[10]}, say, then f[5] would evaluate to f[10]. The recursive pure function is used as a path compressor (to set up memoization in such a way that instead of long chains like f[1]=f[3],f[3]=f[4],f[4]=f[2], ..., memoized values get corrected whenever a new "root" of the component is discovered. This gives a significant speed-up. Because we use assignment, we need it to be HoldAll, which makes this construct even more obscure and more attractive ). This function is a result of on and off-line Mathgroup discussion involving Fred Simons, Szabolcs Horvat, DrMajorBob and yours truly. Example:

In[13]:= largeTest=RandomInteger[{1,80000},{40000,2}];

In[14]:= componentsBFLS[largeTest]//Short//Timing
Out[14]= {0.828,{{33686,62711,64315,11760,35384,45604,10212,52552,63986,  
     <<8>>,40962,7294,63002,38018,46533,26503,43515,73143,5932},<<10522>>}}

It is certainly much slower than a built-in, but for the size of code, quite fast still IMO.

Another example: here is a recursive realization of Select, based on linked lists and recursive pure functions:

selLLNaive[x_List, test_] :=
  Flatten[If[TrueQ[test[#1]],
     {#1, If[#2 === {}, {}, #0 @@ #2]},
     If[#2 === {}, {}, #0 @@ #2]] & @@ Fold[{#2, #1} &, {}, Reverse[x]]];

For example,

In[5]:= Block[
         {$RecursionLimit= Infinity},
         selLLNaive[Range[3000],EvenQ]]//Short//Timing

Out[5]= {0.047,{2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,
 <<1470>>,2972,2974,2976,2978,2980,2982,2984,2986,2988,2990,
  2992,2994,2996,2998,3000}}

It is however not properly tail recursive, and will blow the stack (crash the kernel) for larger lists. Here is the tail-recursive version:

selLLTailRec[x_List, test_] :=
Flatten[ If[Last[#1] === {},
  If[TrueQ[test[First[#1]]],
   {#2, First[#1]}, #2],
  (* else *)
  #0[Last[#1],
   If[TrueQ[test[First[#1]]], {#2, First[#1]}, #2]
   ]] &[Fold[{#2, #1} &, {}, Reverse[x]], {}]];

For example,

In[6]:= Block[{$IterationLimit= Infinity},
       selLLTailRec[Range[500000],EvenQ]]//Short//Timing
Out[6]= {2.39,{2,4,6,8,10,12,14,16,18,20,22,
       <<249978>>,499980,499982,499984,499986,499988,499990,499992,
        499994,499996,499998,500000}} 

This is recipe from Stan Wagon's book...use it when built-in Plot behaves erratically due to lack of precision

Options[PrecisePlot] = {PrecisionGoal -> 6};
PrecisePlot[f_, {x_, a_, b_}, opts___] := Module[{g, pg},
   pg = PrecisionGoal /. {opts} /. Options[PrecisePlot];
   SetAttributes[g, NumericFunction];
   g[z_?InexactNumberQ] := Evaluate[f /. x -> z];
   Plot[N[g[SetPrecision[y, \[Infinity]]], pg], {y, a, b},
    Evaluate[Sequence @@ FilterRules[{opts}, Options[Plot]]]]];

I often use the following trick from Kristjan Kannike's when I need "dictionary-like" behavior from Mathematica's downvalues

index[downvalue_, 
   dict_] := (downvalue[[1]] /. HoldPattern[dict[x_]] -> x) // 
   ReleaseHold;
value[downvalue_] := downvalue[[-1]];
indices[dict_] := 
  Map[#[[1]] /. {HoldPattern[dict[x_]] -> x} &, DownValues[dict]] // 
   ReleaseHold;
values[dict_] := Map[#[[-1]] &, DownValues[dict]];
items[dict_] := Map[{index[#, dict], value[#]} &, DownValues[dict]];
indexQ[dict_, index_] := 
  If[MatchQ[dict[index], HoldPattern[dict[index]]], False, True];

(* Usage example: *)
(* Count number of times each subexpression occurs in an expression *)
expr = Cos[x + Cos[Cos[x] + Sin[x]]] + Cos[Cos[x] + Sin[x]]
Map[(counts[#] = If[indexQ[counts, #], counts[#] + 1, 1]; #) &, expr, Infinity];
items[counts]

When evaluation results are confusing, sometimes it helps to dump evaluation steps into a text file

SetAttributes[recordSteps, HoldAll];
recordSteps[expr_] :=
 Block[{$Output = List@OpenWrite["~/temp/msgStream.m"]}, 
  TracePrint[Unevaluated[expr], _?(FreeQ[#, Off] &), 
   TraceInternal -> True];
  Close /@ $Output;
  Thread[Union@    Cases[ReadList["~/temp/msgStream.m", HoldComplete[Expression]], 
     symb_Symbol /; 
       AtomQ@Unevaluated@symb && 
        Context@Unevaluated@symb === "System`" :> 
      HoldComplete@symb, {0, Infinity}, Heads -> True], HoldComplete]
  ]

(* Usage example: *)
(* puts steps of evaluation of 1+2+Sin[5]) into ~/temp/msgStream.m *)
recordSteps[1+2+Sin[5]]

It is possible to run MathKernel in batch mode by using undocumented command-line options -batchinput and -batchoutput:

math -batchinput -batchoutput < input.m > outputfile.txt

(where input.m is the batch input file ending with the newline character, outputfile.txt is the file to which the output will be redirected).

In Mathematica v.>=6 the MathKernel has undocumented command-line option:

-noicon

which controls whether the MathKernel will have visible icon on the Taskbar (at least under Windows).

The FrontEnd (at least from v.5) has undocumented command-line option

-b

which disables the splash-screen and allows to run the Mathematica FrontEnd much faster

and option

-directlaunch

which disables the mechanism which launches the most recent Mathematica version installed instead of launching the version associated with .nb files in the system registry.

Another way to do this probably is:

> Instead of launching the > Mathematica.exe binary in the > installation directory, launch the > Mathematica.exe binary in > SystemFiles\FrontEnd\Binaries\Windows. > The former is a simple launcher > program which tries its hardest to > redirect requests for opening > notebooks to running copies of the > user interface. The latter is the > user interface binary itself.

It is handy to combine the last command line option with setting global FrontEnd option VersionedPreferences->True which disables sharing of preferences between different Mathematica versions installed:

SetOptions[$FrontEnd, VersionedPreferences -> True]

(The above should be evaluated in the most recent Mathematica version installed.)

In Mathematica 8 this is controlled in the Preferences dialog, in the System pane, under the setting "Create and maintain version specific front end preferences".

It is possible to get incomplete list of command-line options of the FrontEnd by using undocumented key -h (the code for Windows):

SetDirectory[$InstallationDirectory <> 
   "\\SystemFiles\\FrontEnd\\Binaries\\Windows\\"];
Import["!Mathematica -h", "Text"]

gives:

Usage:  Mathematica [options] [files]
Valid options:
	-h (--help):  prints help message
	-cleanStart (--cleanStart):  removes existing preferences upon startup
	-clean (--clean):  removes existing preferences upon startup
	-nogui (--nogui):  starts in a mode which is initially hidden
	-server (--server):  starts in a mode which disables user interaction
	-activate (--activate):  makes application frontmost upon startup
	-topDirectory (--topDirectory):  specifies the directory to search for resources and initialization files
	-preferencesDirectory (--preferencesDirectory):  specifies the directory to search for user AddOns and preference files
	-password (--password):  specifies the password contents
	-pwfile (--pwfile):  specifies the path for the password file
	-pwpath (--pwpath):  specifies the directory to search for the password file
	-b (--b):  launches without the splash screen
	-min (--min):  launches as minimized

Other options include:

-directLaunch:  force this FE to start
-32:  force the 32-bit FE to start
-matchingkernel:  sets the frontend to use the kernel of matching bitness
-Embedding:  specifies that this instance is being used to host content out of process

---

Are there other potentially useful command-line options of the MathKernel and the FrontEnd? Please share if you know.

Related question.

My favorite hacks are small code-generating macros that allow you to replace a bunch of standard boilerplate commands with one short one. Alternatively, you can create commands for opening/creating notebooks.

Here is what I've been using for a while in my day-to-day Mathematica workflow. I found myself performing the following a lot:

1. Make a notebook have a private context, load package(s) I need, make it autosave.
2. After working with this notebook for a while, I'd want to do some throw away scratch computations in a separate notebook, with its own private context, while having access to definitions I've been using in the "main" notebook. Because I set up the private context, this requires to manually adjust $ContextPath

Doing all this by hand over and over is a pain, so let's automate! First, some utility code:

(* Credit goes to Sasha for SelfDestruct[] *)
SetAttributes[SelfDestruct, HoldAllComplete];
SelfDestruct[e_] := (If[$FrontEnd =!= $Failed,
   SelectionMove[EvaluationNotebook[], All, EvaluationCell]; 
   NotebookDelete[]]; e)

writeAndEval[nb_,boxExpr_]:=(
    NotebookWrite[nb,  CellGroupData[{Cell[BoxData[boxExpr],"Input"]}]];
    SelectionMove[nb, Previous, Cell]; 
    SelectionMove[nb, Next, Cell];
    SelectionEvaluate[nb];
)

ExposeContexts::badargs = 
  "Exposed contexts should be given as a list of strings.";
ExposeContexts[list___] := 
 Module[{ctList}, ctList = Flatten@List@list; 
  If[! MemberQ[ctList, Except[_String]],AppendTo[$ContextPath, #] & /@ ctList, 
   Message[ExposeContexts::badargs]];
  $ContextPath = DeleteDuplicates[$ContextPath];
  $ContextPath]

    Autosave[x:(True|False)] := SetOptions[EvaluationNotebook[],NotebookAutoSave->x];

Now, let's create a macro that's going to put the following cells in the notebook:

SetOptions[EvaluationNotebook[], CellContext -> Notebook]
Needs["LVAutils`"]
Autosave[True]

And here's the macro:

MyPrivatize[exposedCtxts : ({__String} | Null) : Null]:=
  SelfDestruct@Module[{contBox,lvaBox,expCtxtBox,assembledStatements,strList},
    contBox = MakeBoxes[SetOptions[EvaluationNotebook[], CellContext -> Notebook]];
    lvaBox = MakeBoxes[Needs["LVAutils`"]];
    
    assembledStatements = {lvaBox,MakeBoxes[Autosave[True]],"(*********)"};
    assembledStatements = Riffle[assembledStatements,"\[IndentingNewLine]"]//RowBox;
    writeAndEval[InputNotebook[],contBox];
    writeAndEval[InputNotebook[],assembledStatements];
    If[exposedCtxts =!= Null,
       strList = Riffle[("\"" <> # <> "\"") & /@ exposedCtxts, ","];
       expCtxtBox = RowBox[{"ExposeContexts", "[", RowBox[{"{", RowBox[strList], "}"}], "]"}];
       writeAndEval[InputNotebook[],expCtxtBox];
      ]
 ]

Now when I type in MyPrivatize[] is creates the private context and loads my standard package. Now let's create a command that will open a new scratch notebook with its own private context (so that you can hack there with wild abandon without the risk of screwing up the definitions), but has access to your current contexts.

SpawnScratch[] := SelfDestruct@Module[{nb,boxExpr,strList},
    strList = Riffle[("\"" <> # <> "\"") & /@ $ContextPath, ","];
    boxExpr = RowBox[{"MyPrivatize", "[",
        RowBox[{"{", RowBox[strList], "}"}], "]"}];
    nb = CreateDocument[];
    writeAndEval[nb,boxExpr];
]

The cool thing about this is that due to SelfDestruct, when the command runs it leaves no trace in the current notebook -- which is good, because otherwise it would just create clutter.

For extra style points, you can create keyword triggers for these macros using InputAutoReplacements, but I'll leave this as an exercise for the reader.

PutAppend with PageWidth -> Infinity

In Mathematica using of the PutAppend command is the most straightforward way to maintain a running log file with results of intermediate computations. But it uses by default PageWith->78 setting when exporting expressions to a file and so there is no guarantee that every intermediate output will take only one line in the log.

PutAppend does not have any options itself but tracing its evaluations reveals that it is based on the OpenAppend function which has the PageWith option and allows changing its default value by the SetOptions command:

In[2]:= Trace[x>>>"log.txt",TraceInternal->True]
Out[2]= {x>>>log.txt,{OpenAppend[log.txt,CharacterEncoding->PrintableASCII],OutputStream[log.txt,15]},Null}

So we can get PutAppend to append only one line at a time by setting:

SetOptions[OpenAppend, PageWidth -> Infinity]

---

UPDATE

There is a bug introduced in version 10 (fixed in version 11.3): SetOptions no longer affects the behavior of OpenWrite and OpenAppend.

A workaround is to implement your own version of PutAppend with explicit PageWidth -> Infinity option:

Clear[myPutAppend]
myPutAppend[expr_, pathtofile_String] :=
 (Write[#, expr]; Close[#];) &[OpenAppend[pathtofile, PageWidth -> Infinity]]

Note that we also may implement it via WriteString as shown in this answer, but in this case it will be necessary to preliminarily convert the expression into the corresponding InputForm via ToString[expr, InputForm].

I was just looking through one of my packages for inclusion in this, and found some messages that I defined that work wonders: Debug::<some name>. By default, they are turned off, so don't produce much overhead. But, I can litter my code with them, and turn them on if I need to figure out exactly how a bit of code is behaving.

One of the things that bothers me about the built-in scoping constructs is that they evaluate all of the local variable definitions at once, so you can't write for example

With[{a = 5, b = 2 * a},    ...]

So a while ago I came up with a macro called WithNest that allows you to do this. I find it handy, since it lets you keep variable bindings local without having to do something like

Module[{a = 5,b},    b = 2 * a;    ...]

In the end, the best way I could find to do this was by using a special symbol to make it easier to recurse over the list of bindings, and I put the definition into its own package to keep this symbol hidden. Maybe someone has a simpler solution to this problem?

If you want to try it out, put the following into a file called Scoping.m:

BeginPackage["Scoping`"];

WithNest::usage=
"WithNest[{var1=val1,var2=val2,...},body] works just like With, except that
values are evaluated in order and later values have access to earlier ones.
For example, val2 can use var1 in its definition.";

Begin["`Private`"];

(* Set up a custom symbol that works just like Hold. *)
SetAttributes[WithNestHold,HoldAll];

(* The user-facing call.  Give a list of bindings and a body that's not
our custom symbol, and we start a recursive call by using the custom
symbol. *)
WithNest[bindings_List,body:Except[_WithNestHold]]:=
WithNest[bindings,WithNestHold[body]];

(* Base case of recursive definition *)
WithNest[{},WithNestHold[body_]]:=body;

WithNest[{bindings___,a_},WithNestHold[body_]]:=
WithNest[
{bindings},
WithNestHold[With[List@a,body]]];

SyntaxInformation[WithNest]={"ArgumentsPattern"->{{__},_}};
SetAttributes[WithNest,{HoldAll,Protected}];

End[];

EndPackage[];

This one was written by Alberto Di Lullo, (who doesn't appear to be on Stack Overflow).

CopyToClipboard, for Mathematica 7 (in Mathematica 8 it's built in)

CopyToClipboard[expr_] := 
  Module[{nb}, 
   nb = CreateDocument[Null, Visible -> False, WindowSelected -> True];
   NotebookWrite[nb, Cell[OutputFormData@expr], All];
   FrontEndExecute[FrontEndToken[nb, "Copy"]];
   NotebookClose@nb];

Original post: http://forums.wolfram.com/mathgroup/archive/2010/Jun/msg00148.html

I have found this routine useful for copying large real numbers to the clipboard in ordinary decimal form. E.g. CopyToClipboard["123456789.12345"]

Cell[OutputFormData@expr] neatly removes the quotes.

This code makes a palette that uploads the selection to Stack Exchange as an image. On Windows, an extra button is provided that gives a more faithful rendering of the selection.

Copy the code into a notebook cell and evaluate. Then pop out the palette from the output, and install it using Palettes -> Install Palette...

If you have any trouble with it, post a comment here. Download the notebook version here.

---

Begin["SOUploader`"];

Global`palette = PaletteNotebook@DynamicModule[{},

   Column[{     Button["Upload to SE",      With[{img = rasterizeSelection1[]},
       If[img === $Failed, Beep[], uploadWithPreview[img]]],
      Appearance -> "Palette"],

     If[$OperatingSystem === "Windows",      Button["Upload to SE (pp)",       With[{img = rasterizeSelection2[]},
        If[img === $Failed, Beep[], uploadWithPreview[img]]],
       Appearance -> "Palette"],

      Unevaluated@Sequence[]
      ]
     }],

   (* Init start *)
   Initialization :>
    (

     stackImage::httperr = "Server returned respose code: `1`";
     stackImage::err = "Server returner error: `1`";

     stackImage[g_] :=
      Module[
       {getVal, url, client, method, data, partSource, part, entity,
        code, response, error, result},

       getVal[res_, key_String] :=
        With[{k = "var " <> key <> " = "},
         StringTrim[          First@StringCases[            First@Select[res, StringMatchQ[#, k ~~ ___] &],
            k ~~ v___ ~~ ";" :> v],
          "'"]
         ];

       data = ExportString[g, "PNG"];

       JLink`JavaBlock[        url = "http://stackoverflow.com/upload/image";        client =         JLink`JavaNew["org.apache.commons.httpclient.HttpClient"];
        method =
         JLink`JavaNew[          "org.apache.commons.httpclient.methods.PostMethod", url];
        partSource =
         JLink`JavaNew[          "org.apache.commons.httpclient.methods.multipart.\
ByteArrayPartSource", "mmagraphics.png",          JLink`MakeJavaObject[data]@toCharArray[]];
        part =
         JLink`JavaNew[          "org.apache.commons.httpclient.methods.multipart.FilePart",          "name", partSource];
        part@setContentType["image/png"];
        entity =
         JLink`JavaNew[          "org.apache.commons.httpclient.methods.multipart.\
MultipartRequestEntity", {part}, method@getParams[]];
        method@setRequestEntity[entity];
        code = client@executeMethod[method];
        response = method@getResponseBodyAsString[];
        ];

       If[code =!= 200, Message[stackImage::httperr, code];
        Return[$Failed]];
       response = StringTrim /@ StringSplit[response, "\n"];

       error = getVal[response, "error"];
       result = getVal[response, "result"];
       If[StringMatchQ[result, "http*"],
        result,
        Message[stackImage::err, error]; $Failed]
       ];

     stackMarkdown[g_] :=
      "![Mathematica graphics](" <> stackImage[g] <> ")";

     stackCopyMarkdown[g_] := Module[{nb, markdown},
       markdown = Check[stackMarkdown[g], $Failed];
       If[markdown =!= $Failed,        nb = NotebookCreate[Visible -> False];
        NotebookWrite[nb, Cell[markdown, "Text"]];
        SelectionMove[nb, All, Notebook];
        FrontEndTokenExecute[nb, "Copy"];
        NotebookClose[nb];
        ]
       ];

     (* Returns available vertical screen space,
     taking into account screen elements like the taskbar and menu *)


     screenHeight[] := -Subtract @@
        Part[ScreenRectangle /. Options[$FrontEnd, ScreenRectangle],
         2];

     uploadWithPreview[img_Image] :=
      CreateDialog[
       Column[{
         Style["Upload image to the Stack Exchange network?", Bold],
         Pane[          Image[img, Magnification -> 1], {Automatic,
           Min[screenHeight[] - 140, 1 + ImageDimensions[img][[2]]]},
          Scrollbars -> Automatic, AppearanceElements -> {},
          ImageMargins -> 0
          ],
         Item[          ChoiceButtons[{"Upload and copy MarkDown"}, \{stackCopyMarkdown[img]; DialogReturn[]}], Alignment -> Right]
         }],
       WindowTitle -> "Upload image to Stack Exchange?"
       ];

     (* Multiplatform, fixed-width version.
        The default max width is 650 to fit Stack Exchange *)
     rasterizeSelection1[maxWidth_: 650] :=
      Module[{target, selection, image},
       selection = NotebookRead[SelectedNotebook[]];
       If[MemberQ[Hold[{}, $Failed, NotebookRead[$Failed]], selection],

        $Failed, (* There was nothing selected *)

        target =
         CreateDocument[{}, WindowSelected -> False, Visible -> False,           WindowSize -> maxWidth];
        NotebookWrite[target, selection];
        image = Rasterize[target, "Image"];
        NotebookClose[target];
        image
        ]
       ];

     (* Windows-only pixel perfect version *)
     rasterizeSelection2[] :=
      If[
       MemberQ[Hold[{}, $Failed, NotebookRead[$Failed]],
        NotebookRead[SelectedNotebook[]]],

       $Failed, (* There was nothing selected *)

       Module[{tag},        FrontEndExecute[         FrontEndToken[FrontEnd`SelectedNotebook[], "CopySpecial",
          "MGF"]];
        Catch[         NotebookGet@ClipboardNotebook[] /.
          r_RasterBox :>
           Block[{},
            Throw[Image[First[r], "Byte", ColorSpace -> "RGB"], tag] /;
              True];
         $Failed,
         tag
         ]
        ]
       ];
     )
   (* Init end *)
   ]

End[];

I'm sure a lot of people have encountered the situation where they run some stuff, realizing it not only stuck the program, but they also haven't saved for the last 10 minutes!

EDIT

After suffering from this for some time, I one day found out that one can create auto-save from within the Mathematica code. I think that using such auto-save have helped me a lot in the past, and I always felt that the possibility itself was something that not a lot of people are aware that they can do.

The original code I used is at the bottom. Thanks to the comments I've found out that it is problematic, and that it is much better to do it in an alternative way, using ScheduledTask (which will work only in Mathematica 8).

Code for this can be found in this answer from Sjoerd C. de Vries (Since I'm not sure if it's OK to copy it to here, I'm leaving it as a link only.)

---

The solution below is using Dynamic. It will save the notebook every 60 seconds, but apparently only if its cell is visible. I'm leaving it here only for completion reasons. (and for users of Mathematica 6 and 7)

/EDIT

To solve it I use this code in the beginning of a notebook:

Dynamic[Refresh[NotebookSave[]; DateString[], UpdateInterval -> 60]]

This will save your work every 60 seconds.
I prefer it to NotebookAutoSave[] because it saves before the input is processed, and because some files are more text than input.

I originally found it here: http://en.wikipedia.org/wiki/Talk:Mathematica#Criticisms

Note that once running this line, saving will happen even if you close and re-open your file (as long as dynamic updating is enabled).

Also, since there is no undo in Mathematica, be careful not to delete all your content, since saving will make it irreversible (as a precaution move, I remove this code from every finished notebook)

Remember that The Mathematica Book is also available online at http://reference.wolfram.com/legacy/v5_2/ - though it's superseded by the current documentation at http://reference.wolfram.com

I find it really useful when developing packages to add this keyboard shortcut to my SystemFiles/FrontEnd/TextResources/Windows/KeyEventTranslations.tr file.

(* Evaluate Initialization Cells: Real useful for reloading library changes. *)

Item[KeyEvent["i", Modifiers -> {Control, Command}],
    FrontEndExecute[
	    FrontEndToken[
		    SelectedNotebook[],
		    "EvaluateInitialization"]]],

Next for every Packagename.m I make a PackagenameTest.nb notebook for testing and the first 2 cells of the test notebook are set as initialization cells. In the first cell I put

Needs["PackageManipulations`"]

to load the very useful PackageManipulations library which was written by Leonid. The second cell contains

PackageRemove["Packagename`Private`"]
PackageRemove["Packagename`"]
PackageReload["Packagename`"]

which all do the actual package reloading. Note the first two lines are there only to Remove all symbols as I like to keep the contexts as clean as possible.

Then the workflow for writing and testing a package becomes something like this.

1. Save changes to Packagename.m.
2. Go to PackagenameTest.nb and do CTRL + ALT + i.

This causes the initialization cells to reload the package, which makes testing real simple.

Following function format[expr_] can be used to indent/format unformatted mathematica expressions that spans over a page

indent[str_String, ob_String, cb_String, delim_String] := 
  Module[{ind, indent, f, tab}, ind = 0; tab = "    ";   indent[i_, tab_, nl_] := nl <> Nest[tab <> ToString[#] &, "", i];
   f[c_] := (indent[ind, "", " "] <> c <> indent[++ind, tab, "\n"]) /;StringMatchQ[ob, ___ ~~ c ~~ ___];
   f[c_] := (indent[--ind, "", " "] <> c <> indent[ind, tab, "\n"]) /;StringMatchQ[cb, ___ ~~ c ~~ ___];
   f[c_] := (c <> indent[ind, tab, "\n"]) /;StringMatchQ[delim, ___ ~~ c ~~ ___];
   f[c_] := c;
   f /@ Characters@str // StringJoin];
format[expr_] := indent[expr // InputForm // ToString, "[({", "])}", ";"];

(*    
format[Hold@Module[{ind, indent, f, tab}, ind = 0; tab = "    ";
 indent[i_, tab_, nl_] := nl <> Nest[tab <> ToString[#] &, "", i];
 f[c_] := (indent[ind, "", " "] <> c <> indent[++ind, tab, "\n"]) /;StringMatchQ[ob, ___ ~~ c ~~ ___];
 f[c_] := (indent[--ind, "", " "] <> c <> indent[ind, tab, "\n"]) /;StringMatchQ[cb, ___ ~~ c ~~ ___];
 f[c_] := (c <> indent[ind, tab, "\n"]) /;StringMatchQ[delim, ___ ~~ c ~~ ___];
 f[c_] := c;
 f /@ Characters@str // StringJoin]]
*)

---

ref: https://codegolf.stackexchange.com/questions/3088/indent-a-string-using-given-parentheses