Continuations can be modelled as monads, using CPS, but they are not algebraic effects as they cannot be modelled using Lawvere theories.  See Martin Hyland and John Power, 2007, [The Category Theoretic Understanding of Universal Algebra: Lawvere Theories and Monads (pdf)](https://www.dpmms.cam.ac.uk/~martin/Research/Publications/2007/hp07.pdf), ENTCS 172:437-458.

I am running a Jenkins cluster where in the Master and Slave, both are running as a Docker containers. 

The Host is latest boot2docker VM running on MacOS.

To allow Jenkins to be able to perform deployment using Docker, I have mounted the docker.sock and docker client from the host to the Jenkins container like this :-

    docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):/usr/bin/docker -v $HOST_JENKINS_DATA_DIRECTORY/jenkins_data:/var/jenkins_home -v $HOST_SSH_KEYS_DIRECTORY/.ssh/:/var/jenkins_home/.ssh/ -p 8080:8080 jenkins

I am facing issues while mounting a volume to Docker containers that are run inside the Jenkins container. For example, if I need to run another Container inside the Jenkins container, I do the following :-

    sudo docker run -v $JENKINS_CONTAINER/deploy.json:/root/deploy.json $CONTAINER_REPO/$CONTAINER_IMAGE 

The above runs the container, but the file &quot;deploy.json&quot; is NOT mounted as a file, but instead as a &quot;Directory&quot;. Even if I mount a Directory as a Volume, I am unable to view the files in the resulting container. 

Is this a problem, because of file permissions due to Docker in Docker case?

 

Docker in Docker cannot mount volume

Trying to make a function that uses filter but not a for or while loop or foreach function, that will loop through an array of objects only to return their property values. For example, 
```js
function getShortMessages(messages) {
    return messages.filter(function(obj){
        return obj.message
    });
}
```
so if I call 
```js
getShortMessages([{message:&quot;bleh&quot;},{message:&quot;blah&quot;}]); 
```
I should get a return of an array = [&quot;bleh&quot;,&quot;blah&quot;] 
I&#39;m just not sure how to implement filter under these guidelines. Also I was thinking of using a chain function maybe .map.

//// Here is the entire code challenge specification/////

Basic: Filter
 Exercise 4 of 18


# Task

Use Array#filter to write a function called getShortMessages.

getShortMessages takes an array of objects with &#39;.message&#39; properties and returns an array of messages that are less than &lt; 50 characters long.

The function should return an array containing the messages themselves, without their containing object.

## Arguments

  * messages: an Array of 10 to 100 random objects that look something like this:
```json
{
    message: &#39;Esse id amet quis eu esse aute officia ipsum.&#39; // random
}
```

## Conditions

  * Do not use any for/while loops or Array#forEach.
  * Do not create any unnecessary functions e.g. helpers.

## Hint

  * Try chaining some Array methods!

## Example

    [ &#39;Tempor quis esse consequat sunt ea eiusmod.&#39;,
      &#39;Id culpa ad proident ad nulla laborum incididunt.&#39;,
      &#39;Ullamco in ea et ad anim anim ullamco est.&#39;,
      &#39;Est ut irure irure nisi.&#39; ]

## Resources

  * https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter
  * https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/map

## Boilerplate

    function getShortMessages(messages) {
      // SOLUTION GOES HERE
    }
    
    module.exports = getShortMessages


 &#187; To print these instructions again, run: functional-javascript print
 &#187; To execute your program in a test environment, run: functional-javascript run program.js
 &#187; To verify your program, run: functional-javascript verify program.js
 &#187; For help run: functional-javascript help


use filter to return property values in an object

The paper [&quot;Programming and reasoning with algebraic effects and dependent types&quot; by Edwin C. Brady][1] on effects in Idris contains the (unreferenced) claim that:

&gt; Although [effects and monad transformers] are not equivalent in power — monads and monad transformers can express more concepts — many common effectful computations are captured.

What examples are there that can be modelled by monad transformers but not effects?

  [1]: http://eb.host.cs.st-andrews.ac.uk/drafts/effects.pdf &quot;Edwin C. Brady (2013?): &#39;Programming and reasoning with algebraic effects and dependent types&#39;&quot;

&quot;Monad transformers more powerful than effects&quot; - Examples?

The paper ["Programming and reasoning with algebraic effects and dependent types" by Edwin C. Brady][1] on effects in Idris contains the (unreferenced) claim that:
> Although [effects and monad transformers] are not equivalent in power — monads and monad transformers can express more concepts — many common effectful computations are captured.
What examples are there that can be modelled by monad transformers but not effects?
[1]: <a href="http://eb.host.cs.st-andrews.ac.uk/drafts/effects.pdf" target="_blank" rel="noopener noreferrer">http://eb.host.cs.st-andrews.ac.uk/drafts/effects.pdf</a> "Edwin C. Brady (2013?): 'Programming and reasoning with algebraic effects and dependent types'"

Is there a better functional way of converting an array of Strings in the form of &quot;key:value&quot; to a `Map` using the Java 8 lambda syntax?

    Arrays.asList(&quot;a:1.0&quot;, &quot;b:2.0&quot;, &quot;c:3.0&quot;)
            .stream()
            .map(elem -&gt; elem.split(&quot;:&quot;)
            .collect(Collectors.toMap(keyMapper?, valueMapper?));

The solution I have right now does not seem really functional:

    Map&lt;String, Double&gt; kvs = new HashMap&lt;&gt;();
    Arrays.asList(&quot;a:1.0&quot;, &quot;b:2.0&quot;, &quot;c:3.0&quot;)
            .stream()
            .map(elem -&gt; elem.split(&quot;:&quot;))
            .forEach(elem -&gt; kvs.put(elem[0], Double.parseDouble(elem[1])));



Convert String array to Map using Java 8 Lambda expressions

Church numbers are an encoding of natural numbers as functions.

    (\ f x → (f x))             -- church number 1
    (\ f x → (f (f (f x))))     -- church number 3
    (\ f x → (f (f (f (f x))))) -- church number 4

Neatly, you can exponentiate 2 church numbers by just applying them. That is, if you apply 4 to 2, you get the church number `16`, or `2^4`. Obviously,  that is utterly unpractical. Church numbers need a linear amount of memory and are really, really slow. Computing something like `10^10` - which GHCI quickly answers correctly - would take ages and couldn&#39;t fit the memory on your computer anyway. 

I&#39;ve been experimenting with optimal λ evaluators lately. On my tests, I accidentally typed the following on my optimal λ-calculator:

    10 ^ 10 % 13

It was supposed to be multiplication, not exponentiation. Before I could move my fingers to abort the forever-running program in despair, it answered my request:
 
    3
    { iterations: 11523, applications: 5748, used_memory: 27729 }

    real	0m0.104s
    user	0m0.086s
    sys	    0m0.019s

With my &quot;bug alert&quot; flashing, I went to Google and verified, `10^10%13 == 3` indeed. **But the λ-calculator wasn&#39;t supposed to find that result, it can barely store 10^10.** I started stressing it, for science. It instantly answered me `20^20%13 == 3`, `50^50%13 == 4`, `60^60%3 == 0`. I had to use [external tools](https://www.mtholyoke.edu/courses/quenell/s2003/ma139/js/powermod.html) to verify those results, since &lt;s&gt;Haskell itself wasn&#39;t able to compute it (due to integer overflow)&lt;/s&gt; (it is if you use Integers not Ints, of course!). Pushing it to its limits, this was the answer to `200^200%31`:

    5
    { iterations: 10351327, applications: 5175644, used_memory: 23754870 }

    real	0m4.025s
    user	0m3.686s
    sys	0m0.341s

If we had one copy of the universe for each atom on the universe, and we had a computer for each atom we had in total, we couldn&#39;t store the church number `200^200`. This prompted me to question if my mac was really that powerful. Maybe the optimal evaluator was able to skip the unnecessary branches and arrive right at the answer in the same fashion Haskell does with lazy evaluation. To test this, I compiled the λ program to Haskell:

    data Term = F !(Term -&gt; Term) | N !Double
    instance Show Term where {
        show (N x) = &quot;(N &quot;++(if fromIntegral (floor x) == x then show (floor x) else show x)++&quot;)&quot;;
        show (F _) = &quot;(λ...)&quot;}
    infixl 0 #
    (F f) # x = f x
    churchNum = F(\(N n)-&gt;F(\f-&gt;F(\x-&gt;if n&lt;=0 then x else (f#(churchNum#(N(n-1))#f#x)))))
    expMod    = (F(\v0-&gt;(F(\v1-&gt;(F(\v2-&gt;((((((churchNum # v2) # (F(\v3-&gt;(F(\v4-&gt;(v3 # (F(\v5-&gt;((v4 # (F(\v6-&gt;(F(\v7-&gt;(v6 # ((v5 # v6) # v7))))))) # v5))))))))) # (F(\v3-&gt;(v3 # (F(\v4-&gt;(F(\v5-&gt;v5)))))))) # (F(\v3-&gt;((((churchNum # v1) # (churchNum # v0)) # ((((churchNum # v2) # (F(\v4-&gt;(F(\v5-&gt;(F(\v6-&gt;(v4 # (F(\v7-&gt;((v5 # v7) # v6))))))))))) # (F(\v4-&gt;v4))) # (F(\v4-&gt;(F(\v5-&gt;(v5 # v4))))))) # ((((churchNum # v2) # (F(\v4-&gt;(F(\v5-&gt;v4))))) # (F(\v4-&gt;v4))) # (F(\v4-&gt;v4))))))) # (F(\v3-&gt;(((F(\(N x)-&gt;F(\(N y)-&gt;N(x+y)))) # v3) # (N 1))))) # (N 0))))))))
    main = print $ (expMod # N 5 # N 5 # N 4)

This correctly outputs `1` (`5 ^ 5 % 4`) - but throw anything above `10^10` and it will be stuck, eliminating the hypothesis. 

The [optimal evaluator I used](https://github.com/SrVictorMaia/optlam) is a 160-lines long, unoptimized JavaScript program that didn&#39;t include any sort of exponential modulus math - and the lambda-calculus modulus function I used was equally simple:

    (λab.(b(λcd.(c(λe.(d(λfg.(f(efg)))e))))(λc.(c(λde.e)))(λc.(a(b(λdef.(d(λg.(egf))))(λd.d)(λde.(ed)))(b(λde.d)(λd.d)(λd.d))))))

I used no specific modular arithmetic algorithm or formula. **So, how is the optimal evaluator able to arrive at the right answers?**





Why are λ-calculus optimal evaluators able to compute big modular exponentiations without formulas?

How to implement &quot;partition&quot; operation on Java 8 Stream? By partition I mean, divide a stream into sub-streams of a given size. Somehow it will be identical to Guava [Iterators.partition()][1] method, just it&#39;s desirable that the partitions are lazily-evaluated Streams rather than List&#39;s.


  [1]:https://google.github.io/guava/releases/20.0/api/docs/com/google/common/collect/Iterators.html#partition-java.util.Iterator-int-

Partition a Java 8 Stream

So, having spend many years in an object oriented world with code reuse, design patterns and best practices always taken into account, I find myself struggling somewhat with code organization and code reuse in world of Spark. 

If I try to write code in a reusable way, it nearly always comes with a performance cost and I end up rewriting it to whatever is optimal for my particular use case. This constant &quot;write what is optimal for this particular use case&quot; also affects code organization, because splitting code into different objects or modules is difficult when &quot;it all really belongs together&quot; and I thus end up with very few &quot;God&quot; object containing long chains of complex transformations. In fact, I frequently think that if I had taken a look at most of the Spark code I&#39;m writing now back when I was working in the object oriented world, I would have winced and dismissed it as &quot;spaghetti code&quot;. 

I have surfed the internet trying to find some sort of equivalent to the best practices of the object oriented world, but without much luck. I can find some &quot;best practices&quot; for functional programming but Spark just adds an extra layer, because performance is such a major factor here. 

So my question to you is, have any of you Spark gurus found some best practices for writing Spark code that you can recommend?

**EDIT**

As written in a comment, I did not actually expect anyone to post an answer on how to _solve_ this problem, but rather I was hoping that someone in this community had come across some Martin Fowler type, who had written som articles or blog posts somewhere on how to address problems with code organization in the world of Spark. 

@DanielDarabos suggested that I might put in an example of a situation where code organization and performance are conflicting. While I find that I frequently have issues with this in my everyday work, I find it a bit hard to boil it down to a good minimal example ;) but I will try.

In the object oriented world, I&#39;m a big fan of the Single Responsibility Principle, so I would make sure that my methods were only responsible for one thing. It makes them reusable and easily testable. So if I had to, say, calculate the sum of some numbers in a list (matching some criteria) and I had to calculate the average of the same number, I would most definitely create two methods - one that calculated the sum and one that calculated the average. Like this:

&lt;!-- language: scala --&gt;
    
    def main(implicit args: Array[String]): Unit = {
      val list = List((&quot;DK&quot;, 1.2), (&quot;DK&quot;, 1.4), (&quot;SE&quot;, 1.5))

      println(&quot;Summed weights for DK = &quot; + summedWeights(list, &quot;DK&quot;)
      println(&quot;Averaged weights for DK = &quot; + averagedWeights(list, &quot;DK&quot;)
    }

    def summedWeights(list: List, country: String): Double = {
      list.filter(_._1 == country).map(_._2).sum
    }

    def averagedWeights(list: List, country: String): Double = {
      val filteredByCountry = list.filter(_._1 == country) 
      filteredByCountry.map(_._2).sum/ filteredByCountry.length
    }


I can of course continue to honor SRP in Spark:


&lt;!-- language: scala --&gt;

    def main(implicit args: Array[String]): Unit = {
      val df = List((&quot;DK&quot;, 1.2), (&quot;DK&quot;, 1.4), (&quot;SE&quot;, 1.5)).toDF(&quot;country&quot;, &quot;weight&quot;)

      println(&quot;Summed weights for DK = &quot; + summedWeights(df, &quot;DK&quot;)
      println(&quot;Averaged weights for DK = &quot; + averagedWeights(df, &quot;DK&quot;)
    }


    def avgWeights(df: DataFrame, country: String, sqlContext: SQLContext): Double = {
      import org.apache.spark.sql.functions._
      import sqlContext.implicits._

      val countrySpecific = df.filter(&#39;country === country)
      val summedWeight = countrySpecific.agg(avg(&#39;weight))

      summedWeight.first().getDouble(0)
    }

    def summedWeights(df: DataFrame, country: String, sqlContext: SQLContext): Double = {
      import org.apache.spark.sql.functions._
      import sqlContext.implicits._

      val countrySpecific = df.filter(&#39;country === country)
      val summedWeight = countrySpecific.agg(sum(&#39;weight))

      summedWeight.first().getDouble(0)
    }

But because my `df` may contain billions of rows I would rather not have to perform the `filter` twice. In fact, performance is directly coupled to EMR cost, so I REALLY don&#39;t want that. To overcome it, I thus decide to violate SRP and simply put the two functions in one and make sure I call persist on the country-filtered `DataFrame`, like this:


&lt;!-- language: scala --&gt;

    def summedAndAveragedWeights(df: DataFrame, country: String, sqlContext: SQLContext): (Double, Double) = {
      import org.apache.spark.sql.functions._
      import sqlContext.implicits._

      val countrySpecific = df.filter(&#39;country === country).persist(StorageLevel.MEMORY_AND_DISK_SER)
      val summedWeights = countrySpecific.agg(sum(&#39;weight)).first().getDouble(0)
      val averagedWeights = summedWeights / countrySpecific.count()

      (summedWeights, averagedWeights)
    }

Now, this example if of course a huge simplification of what&#39;s encountered in real life. Here I could simply solve it by filtering and persisting `df` _before_ handing it to the sum and avg functions (which would also be more SRP), but in real life there may be a number of intermediate calculations going on that are needed again and again. In other words, the `filter` function here is merely an attempt to make a _simple_ example of something that will benefit from being persisted. In fact, I think calls to `persist` is a keyword here. Calling `persist` will vastly speed up my job, but the cost is that I have to tightly couple all code that depends on the persisted `DataFrame` - even if they are logically separate.   

Spark code organization and best practices

I&#39;m learning javascript FP by reading DrBoolean&#39;s [book][1].  

I searched around for functional programming library. I found Ramda and Folktale. Both claim to be functional programming library.

But they are so different:  

 - [Ramda][2] seems to contain utility functions for dealing with list: map, reduce, filter and pure functions: curry, compose. It doesn&#39;t contain anything to deal with monad, functor.  

 - [Folktale][3] however doesn&#39;t contain any utility for list or functions. It seems to implement the some algebraic structures in javascript like monad: Maybe, Task...

Actually I found more libraries, they all seem fall into the two categories. Underscore and lodash are like Ramda. Fantasy-land, pointfree-fantasy are like folktale.

Can these very different libraries both be called _functional_, and if so, what makes each one a functional library? 


  [1]: https://drboolean.gitbooks.io/mostly-adequate-guide
  [2]: http://ramdajs.com/
  [3]: http://folktale.origamitower.com/

why are folktale and ramda so different?

I am trying to define a new monad and I am getting a strange error

**newmonad.hs**
&lt;pre&gt;
newtype Wrapped a = Wrap {unwrap :: a}
instance Monad Wrapped where
  (&gt;&gt;=) (Wrap x) f =  f x
  return x = Wrap x

main = do
  putStrLn &quot;yay&quot;
&lt;/pre&gt;

&lt;pre&gt;
$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.1

$ ghc newmonad.hs 
[1 of 1] Compiling Main             ( newmonad.hs, newmonad.o )

newmonad.hs:2:10:
    No instance for (Applicative Wrapped)
      arising from the superclasses of an instance declaration
    In the instance declaration for ‚&#196;&#242;Monad Wrapped‚&#196;&#244;
&lt;/pre&gt;

Why do I need to define an instance of `Applicative`?

Defining a new monad in haskell raises no instance for Applicative

Could someone give a super simple (few lines) monad transformer example, which is non-trivial (i.e. not using the Identity monad - that I understand).

For example, how would someone create a monad that does IO and can handle failure (Maybe)? 

What would be the simplest example that would demonstrate this?

I have skimmed through a few monad transformer tutorials and they all seem to use State Monad or Parsers or something complicated (for a newbee). I would like to see something simpler than that. I think IO+Maybe would be simple, but I don&#39;t really know how to do that myself.

How could I use an IO+Maybe monad stack?
What would be on top? What would be on bottom? Why?


In what kind of use case would one want to use the IO+Maybe monad or the Maybe+IO monad? Would that make sense to create such a composite monad at all? If yes, when, and why?




Simplest non-trivial monad transformer example for &quot;dummies&quot;, IO+Maybe

I have been studying Haskell for several weeks now (just for fun) and just watched Brian Beckman&#39;s great [video introducing monads](https://www.youtube.com/watch?v=ZhuHCtR3xq8).  He motivates monads with the need to create a more general composition operator.  Following this line of thought, if I have two functions:
```
f :: a -&gt; b
g :: b -&gt; c
```
the composition operator should satisfy
```
h = g . f :: a -&gt; c
```
and from this I can infer the correct type of the `.` operator:
```
(.) : (b -&gt; c) -&gt; (a -&gt; b) -&gt; (a -&gt; c)
```

When it comes to monads, suppose I have two functions:
```
f :: a -&gt; m b
g :: b -&gt; m c
```
It seems to me that the natural choice would have been to define a generalized composition operator that works as follows:
```
h = f &gt;&gt;= g :: a -&gt; m c
```
in which case the `&gt;&gt;=` operator would have a type signature of:
```
(&gt;&gt;=) :: (a -&gt; m b) -&gt; (b -&gt; m c) -&gt; (a -&gt; m c)
```
But actually the operator seems to be defined so that
```
h a = (f a) &gt;&gt;= g :: m c
```
and thus
```
(&gt;&gt;=) : m b -&gt; (b -&gt; m c) -&gt; m c
```
Could someone explain the reasoning behind this choice of definition for bind?  I assume there is some simple connection between the two choices where one can be expressed in terms of the other, but I&#39;m not seeing it at the moment.

Why is the bind operator (&gt;&gt;=) defined as it is?

I&#39;m wondering what is the best way to express smart contracts in typed languages such as Haskell or Idris (so you could, for example, compile it to run on the Ethereum network). My main concern is: what is the type that captures everything that a contract could do?

### Naive solution: EthIO

A naive solution would be to define a contract as a member of an `EthIO` type. Such type would be like Haskell&#39;s `IO`, but instead of enabling system calls, it would include blockchain calls, i.e., it would enable reading from and writing to the blockchain&#39;s state, calling other contracts, getting block data and so on.

    -- incrementer.contract
    
    main: EthIO
    main = do
       x &lt;- SREAD 0x123456789ABCDEF
       SSTORE (x + 1) 0x123456789ABCDEF

This is clearly sufficient to implement any contract, but:

 1. Would be too powerful.

 2. Would be very coupled to the Ethereum blockchain specifically.


### Conservative solution: event sourcing pattern

Under that idea, a contract would be defined as fold over a list of actions:

    type Contract action state = {
        act  : UserID -&gt; action -&gt; state -&gt; state,
        init : state
    }

So, a program would look like:
    
    incrementer.contract

    main : Contract
    main = {
        act _ _ state = state + 1,
        init          = 0
    }

That is, you define an initial state, a type of action, and how that state changes when a user submits an action. That would allow one to define any arbitrary contract that doesn&#39;t involve sending/receiving money. Most blockchains have some kind of currency and most useful contracts involve money somehow, so that type would be way too restrictive.

### Less conservative solution: events + currency

We can make the type above aware of currencies by hardcoding a currency logic into the type above. We&#39;d, thus, get something like:

    type Contract action state = {
        act        : UserID -&gt; action -&gt; state -&gt; state,
        init       : state,
        deposit    : UserID -&gt; Amount -&gt; state -&gt; state,
        withdrawal : UserID -&gt; Amount -&gt; state -&gt; Maybe state
    }

I.e., the contract developer would need to explicitly define how to deal with monetary deposits and withdrawals. That type would be enough to define any self-contained contract which can interact with the host blockchain&#39;s currency. Sadly, **such a contract wouldn&#39;t be able to interact with other contracts**. In practice, contracts often interact with each other. An Exchange, for example, needs to communicate with its exchanged Token contracts to query balances and so on.

### Generalization: global state?

So, let&#39;s take a step back and rewrite the conservative solution as this:

    type Contract = {
        act  : UserID -&gt; Action -&gt; Map ContractID State -&gt; State,
        init : State
    }

Under this definition, the `act` function would have access not only to the contract&#39;s own state but the state of every other contract on the same blockchain. Since every contract can read each other&#39;s state, one could easily implement a communication protocol on top of this, and, thus, such type is sufficient to implement arbitrarily interacting contracts. Also, if the blockchain&#39;s currency was itself implemented as a contract (possibly using a wrapper), then that type would also be sufficient to deal with money, despite not having it hardcoded on the type. But that solution has 2 problems:

1. Peeking at the other contract&#39;s state looks like a very &quot;hacky&quot; way to enable communication;

2. A contract defined this way wouldn&#39;t be able to interact with existing contracts which aren&#39;t aware of that solution. 

### What now?

Now I&#39;m in the dark. I know I&#39;m not in the right abstraction for this problem, but I&#39;m not sure what it would be. *It looks like the root of the problem is that I&#39;m not able to capture the phenomenon of cross-contract communications properly.* What concrete type would be more suitable to define arbitrary smart-contracts?

Content Type	Original Author	Original Content on Stackoverflow
Question	geoff_h	View Question on Stackoverflow
Solution 1 - Functional Programming	Charles Stewart	View Answer on Stackoverflow

"Monad transformers more powerful than effects" - Examples?

Functional Programming Problem Overview

Functional Programming Solutions

Solution 1 - Functional Programming

use filter to return property values in an object

Docker in Docker cannot mount volume

Attributions