From the perspective of someone who has written asyncio code but is looking to better understand the inner workings, what is `yield from`, `await` and how are those useful for allowing asynchronous code?

There is one [highly upvoted](https://stackoverflow.com/questions/9708902/in-practice-what-are-the-main-uses-for-the-new-yield-from-syntax-in-python-3) question asking about the uses of the `yield from` syntax and [one explaining async and await](https://stackoverflow.com/questions/31291129/pep-0492-python-3-5-async-keyword), but both go in depth about different topics and are not really a concise explanation of the underlying code and how it fits in with asyncio.


What does the &quot;yield from&quot; syntax do in asyncio and how is it different from &quot;await&quot;

I am calling suspended function from onCreate(...)

    override fun onCreate(savedInstanceState: Bundle?) {
        ...
        ...
        callGetApi()
    }

and the suspended function is:-

    suspend fun callGetApi() {....}

But the error shows up *Suspend function &#39;callGetApi&#39; should be called only from a coroutine or another suspend function*

Suspend function &#39;callGetApi&#39; should be called only from a coroutine or another suspend function

The concept of a [coroutine][1] sounds very interesting, but I don&#39;t know, if it makes sense in a real productive environment? What are use cases for coroutines, where the coroutine implementation is more elegant, simpler or more efficient than other methods?


  [1]: http://en.wikipedia.org/wiki/Coroutine

What are use cases for coroutines?

    #!/usr/bin/env python3
    # -*- coding: utf-8 -*-
    import time
    
    async def foo():
      await time.sleep(1)
    
    foo()

I couldn&#39;t make this dead simple example to run:

    RuntimeWarning: coroutine &#39;foo&#39; was never awaited foo()

How to use async/await in Python 3.5?

Kotlin corutines is sugar for finite state machine and some task runner (for example, default ForkJoinPool). https://github.com/Kotlin/kotlin-coroutines/blob/master/kotlin-coroutines-informal.md#implementation-details 

In other words, there is no runtime coroutines in java/kotlin runtime yet (but this can change with http://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.html ). Kotlin coroutine is just sequential of tasks, which are executed one by one. Each task can be executed in any thread from thread pool.

Go runtime supports &quot;coroutines&quot;. But goroutines is not the real coroutines. Goroutines does not allow to set yield points in program. Also, Go does not allow to set custom thread pool. You can to set only size of threads in default pool. 

First difference between kotlin coroutines and goroutines is Go runtime manages which coroutine is running at this moment. When goroutine are blocked at some IO operation (or synchronization primitives), Go choices next Job to execute it. In JVM there is no intellectual job switching in such terms. 

Because of this, Go can cheaply change currently running job. Go has only to change few registries &lt;a href=&quot;https://groups.google.com/forum/#!msg/golang-nuts/j51G7ieoKh4/wxNaKkFEfvcJ&quot;&gt;https://groups.google.com/forum/#!msg/golang-nuts/j51G7ieoKh4/wxNaKkFEfvcJ&lt;/a&gt;. But some people say, that JVM can use stack of threads instead of using registers. So there is no saving and loading of registers at all.

The second difference between kotlin coroutines and goroutines is type of coroutines. Kotlin coroutines is stackless coroutines. Goroutines are stackful coroutines. All state of Kotlin coroutines are stored in Kotlin context, which is stored in heap. Goroutines state is stored in registers and thread stack.

I want to know, which coroutines (goroutines and kotlin coroutines) are faster in IO bound tasks? CPU bound tasks? What about memory consumption?

Which of coroutines (goroutines and kotlin coroutines) are faster?

The combination of coroutines and resource acquisition seems like it could have some unintended (or unintuitive) consequences.

The basic question is whether or not something like this works:

    def coroutine():
        with open(path, &#39;r&#39;) as fh:
            for line in fh:
                yield line

Which it does. (You can test it!)

The deeper concern is that `with` is supposed to be something an alternative to `finally`, where you ensure that a resource is released at the end of the block. Coroutines can suspend and resume execution from *within* the `with` block, so **how is the conflict resolved?**

For example, if you open a file with read/write both inside and outside a coroutine while the coroutine hasn&#39;t yet returned:

    def coroutine():
        with open(&#39;test.txt&#39;, &#39;rw+&#39;) as fh:
            for line in fh:
                yield line
    
    a = coroutine()
    assert a.next() # Open the filehandle inside the coroutine first.
    with open(&#39;test.txt&#39;, &#39;rw+&#39;) as fh: # Then open it outside.
        for line in fh:
            print &#39;Outside coroutine: %r&#39; % repr(line)
    assert a.next() # Can we still use it?

&lt;h3&gt;Update&lt;/h3&gt;

I was going for write-locked file handle contention in the previous example, but since most OSes allocate filehandles per-process there will be no contention there. (Kudos to @Miles for pointing out the example didn&#39;t make too much sense.) Here&#39;s my revised example, which shows a real deadlock condition:

    import threading
    
    lock = threading.Lock()
    
    def coroutine():
        with lock:
            yield &#39;spam&#39;
            yield &#39;eggs&#39;
    
    generator = coroutine()
    assert generator.next()
    with lock: # Deadlock!
        print &#39;Outside the coroutine got the lock&#39;
    assert generator.next()

Is it safe to yield from within a &quot;with&quot; block in Python (and why)?

I&#39;m trying to update a list inside the adapter using async, I can see there is too much boilerplate. 

Is it the right way to use Kotlin Coroutines?

can this be optimized more?


    fun loadListOfMediaInAsync() = async(CommonPool) {
            try {
                //Long running task 
                adapter.listOfMediaItems.addAll(resources.getAllTracks())
                runOnUiThread {
                    adapter.notifyDataSetChanged()
                    progress.dismiss()
                }
            } catch (e: Exception) {
                e.printStackTrace()
                runOnUiThread {progress.dismiss()}
            } catch (o: OutOfMemoryError) {
                o.printStackTrace()
                runOnUiThread {progress.dismiss()}
            }
        }

Kotlin Coroutines the right way in Android

What is a coroutine? How are they related to concurrency?

What is a coroutine?

What are the differences between a &quot;coroutine&quot; and a &quot;thread&quot;?


Difference between a &quot;coroutine&quot; and a &quot;thread&quot;?

In the `kotlinx.coroutines` library you can start new coroutine using either `launch` (with `join`) or `async` (with `await`). What is the difference between them? 

What is the difference between launch/join and async/await in Kotlin coroutines

What is the difference between a thread and a fiber? I&#39;ve heard of fibers from ruby and I&#39;ve read heard they&#39;re available in other languages, could somebody explain to me in simple terms what is the difference between a thread and a fiber.

What is the difference between a thread and a fiber?

I was wondering if there&#39;s any library for asynchronous method calls in [Python][1]. It would be great if you could do something like

    @async
    def longComputation():
        &lt;code&gt;
    
    
    token = longComputation()
    token.registerCallback(callback_function)
    # alternative, polling
    while not token.finished():
        doSomethingElse()
        if token.finished():
            result = token.result()


Or to call a non-async routine asynchronously

    def longComputation()
        &lt;code&gt;
    
    token = asynccall(longComputation())


It would be great to have a more refined strategy as native in the language core. Was this considered?

  [1]: http://en.wikipedia.org/wiki/Python_%28programming_language%29


Asynchronous method call in Python?

What is the difference between a coroutine and a continuation and a generator ?

Coroutine vs Continuation vs Generator

I am new to gevents and greenlets. I found some good documentation on how to work with them, but none gave me justification on how and when I should use greenlets!

* What are they really good at?
* Is it a good idea to use them in a proxy server or not?
* Why not threads?

What I am not sure about is how they can provide us with concurrency if they&#39;re basically co-routines.

Greenlet Vs. Threads

I understand the principle of coroutines. I know how to get the standard `StartCoroutine` / `yield return` pattern to work in C# in Unity, e.g. invoke a method  returning `IEnumerator` via `StartCoroutine` and in that method do something, do `yield return new WaitForSeconds(1);` to wait a second, then do something else.

My question is: what&#39;s really going on behind the scenes? What does `StartCoroutine` really do? What `IEnumerator` is `WaitForSeconds` returning? How does `StartCoroutine` return control to the &quot;something else&quot; part of the called method? How does all this interact with Unity&#39;s concurrency model (where lots of things are going on at the same time without use of coroutines)?

How does StartCoroutine / yield return pattern really work in Unity?

I&#39;ve got some example Python code that I need to mimic in C++. I do not require any specific solution (such as co-routine based yield solutions, although they would be acceptable answers as well), I simply need to reproduce the semantics in some manner.

Python
------

This is a basic sequence generator, clearly too large to store a materialized version.

    def pair_sequence():
        for i in range(2**32):
            for j in range(2**32):
                yield (i, j)

The goal is to maintain two instances of the sequence above, and iterate over them in semi-lockstep, but in chunks. In the example below the `first_pass` uses the sequence of pairs to initialize the buffer, and the `second_pass` regenerates the *same exact sequence* and processes the buffer again.

    def run():
        seq1 = pair_sequence()
        seq2 = pair_sequence()

        buffer = [0] * 1000
        first_pass(seq1, buffer)
        second_pass(seq2, buffer)
        ... repeat ...

C++
---

The only thing I can find for a solution in C++ is to mimic `yield` with C++ coroutines, but I haven&#39;t found any good reference on how to do this. I&#39;m also interested in alternative (non general) solutions for this problem. I do not have enough memory budget to keep a copy of the sequence between passes.

Equivalent C++ to Python generator pattern

What are coroutines in [tag:C++20]?

In what ways it is different from &quot;Parallelism2&quot; or/and &quot;Concurrency2&quot; (look into below image)?

The below image is from ISOCPP.

https://isocpp.org/files/img/wg21-timeline-2017-03.png


[![enter image description here][1]][1]


  [1]: https://i.stack.imgur.com/4MtgA.png

What are coroutines in C++20?

I&#39;ve seen several basic Python 3.5 tutorials on asyncio doing the same operation in various flavours.
In this code:

    import asyncio  

    async def doit(i):
        print(&quot;Start %d&quot; % i)
        await asyncio.sleep(3)
        print(&quot;End %d&quot; % i)
        return i
    
    if __name__ == &#39;__main__&#39;:
        loop = asyncio.get_event_loop()
        #futures = [asyncio.ensure_future(doit(i), loop=loop) for i in range(10)]
        #futures = [loop.create_task(doit(i)) for i in range(10)]
        futures = [doit(i) for i in range(10)]
        result = loop.run_until_complete(asyncio.gather(*futures))
        print(result)

All the three variants above that define the ``futures`` variable achieve the same result; the only difference I can see is  that with the third variant the execution is out of order (which should not matter in most cases). Is there any other difference? Are there cases where I can&#39;t just use the simplest variant (plain list of coroutines)?

asyncio.ensure_future vs. BaseEventLoop.create_task vs. simple coroutine?

Can Kotlin coroutines be used in production, and what does their experimental status mean?


Can &quot;experimental&quot; Kotlin coroutines be used in production?

**Background:**

I&#39;m asking this because I currently have an application with many (hundreds to thousands) of threads. Most of those threads are idle a great portion of the time, waiting on work items to be placed in a queue. When a work item comes available, it is then processed by calling some arbitrarily-complex existing code. On some operating system configurations, the application bumps up against kernel parameters governing the maximum number of user processes, so I&#39;d like to experiment with means to reduce the number of worker threads. 

**My proposed solution:**

It seems like a coroutine-based approach, where I replace each worker thread with a coroutine, would help to accomplish this. I can then have a work queue backed by a pool of actual (kernel) worker threads. When an item is placed in a particular coroutine&#39;s queue for processing, an entry would be placed into the thread pool&#39;s queue. It would then resume the corresponding coroutine, process its queued data, and then suspend it again, freeing up the worker thread to do other work.

**Implementation details:**

In thinking about how I would do this, I&#39;m having trouble understanding the functional differences between stackless and stackful coroutines. I have some experience using stackful coroutines using the [Boost.Coroutine](http://www.boost.org/doc/libs/1_57_0/libs/coroutine/doc/html/index.html) library. I find it&#39;s relatively easy to comprehend from a conceptual level: for each coroutine, it maintains a copy of the CPU context and stack, and when you switch to a coroutine, it switches to that saved context (just like a kernel-mode scheduler would).

What is less clear to me is how a stackless coroutine differs from this. In my application, the amount of overhead associated with the above-described queuing of work items is very important. Most implementations that I&#39;ve seen, like [the new CO2 library](https://github.com/jamboree/co2) suggest that stackless coroutines provide much lower-overhead context switches. 

Therefore, I&#39;d like to understand the functional differences between stackless and stackful coroutines more clearly. Specifically, I think of these questions:

- [References like this one](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3708.pdf) suggest that the distinction lies in where you can yield/resume in a stackful vs. stackless coroutine. Is this the case? Is there a simple example of something that I can do in a stackful coroutine but not in a stackless one?

- Are there any limitations on the use of automatic storage variables (i.e. variables &quot;on the stack&quot;)? 

- Are there any limitations on what functions I can call from a stackless coroutine?

- If there is no saving of stack context for a stackless coroutine, where do automatic storage variables go when the coroutine is running?

How do stackless coroutines differ from stackful coroutines?

I doubt it can be done portably, but are there any solutions out there?  I think it could be done by creating an alternate stack and reseting SP,BP, and IP on function entry, and having yield save IP and restore SP+BP.  Destructors and exception safety seem tricky but solvable.

Has it been done?  Is it impossible?

How do you implement Coroutines in C++

Monads can do many amazing, crazy things. They can create variables which hold a superposition of values. They can allow you to access data from the future before you compute it. They can allow you to write destructive updates, but not really. And then the continuation monad allows you to _break people&#39;s minds!_ Ususally your own. ;-)

But here&#39;s a challenge: Can you make a monad which can be _paused_?

&lt;pre&gt;
data Pause s x
instance Monad (Pause s)
mutate :: (s -&gt; s) -&gt; Pause s ()
yield :: Pause s ()
step :: s -&gt; Pause s () -&gt; (s, Maybe (Pause s ()))
&lt;/pre&gt;

The `Pause` monad is a kind of state monad (hence `mutate`, with the obvious semantics). Normally a monad like this has some sort of &quot;run&quot; function, which runs the computation and hands you back the final state. But `Pause` is different: It provides a `step` function, which runs the computation until it calls the magical `yield` function. Here the computation is paused, returning to the caller enough information to resume the computation later.

For extra awesomness: Allow the caller to modify the state between `step` calls. (The type signatures above ought to allow this, for example.)

&lt;hr/&gt;

Use case: It&#39;s often easy to write code that does something complex, but a total PITA to transform it to also _output_ the intermediate states in its operation. If you want the user to be able to _change_ something mid-way through execution, things get complex really fast.

Implementation ideas:

* _Obviously_ it can be done with threads, locks and `IO`. But can we do better? ;-)

* Something insane with a continuation monad?

* Maybe some kind of writer monad, where `yield` just logs the current state, and then we can &quot;pretend&quot; to `step` it by iterating over the states in the log. (Obviously this precludes altering the state between steps, since we&#39;re not really &quot;pausing&quot; anything now.)


The Pause monad

Lets say I have two functions:

    def foo():
      return &#39;foo&#39;

    def bar():
      yield &#39;bar&#39;

The first one is a normal function, and the second is a generator function. Now I want to write something like this:

    def run(func):
      if is_generator_function(func):
         gen = func()
         gen.next()
         #... run the generator ...
      else:
         func()
         
What will a straightforward implementation of `is_generator_function()` look like? Using the `types` package I can test if  `gen` is a generator, but I wish to do so before invoking `func()`. 

Now consider the following case:

    def goo():
      if False:
         yield
      else:
         return

An invocation of `goo()` will return a generator. I presume that the python parser knows that the `goo()` function has a yield statement, and I wonder if it possible to get that information easily.

Thanks!

In python is there a way to check if a function is a &quot;generator function&quot; before calling it?

After reading Eli Bendersky&#39;s article [on implementing state machines via Python coroutines](http://eli.thegreenplace.net/2009/08/29/co-routines-as-an-alternative-to-state-machines/) I wanted to...

- see his example run under Python3
- and also add the appropriate type annotations for the generators

I succeeded in doing the first part (*but without using `async def`s or `yield from`s, I basically just ported the code - so any improvements there are most welcome*). 

But I need some help with the type annotations of the coroutines:

    #!/usr/bin/env python3

    from typing import Callable, Generator

    def unwrap_protocol(header: int=0x61,
                        footer: int=0x62,
                        dle: int=0xAB,
                        after_dle_func: Callable[[int], int]=lambda x: x,
                        target: Generator=None) -&gt; Generator:
        &quot;&quot;&quot; Simplified protocol unwrapping co-routine.&quot;&quot;&quot;
        #
        # Outer loop looking for a frame header
        #
        while True:
            byte = (yield)
            frame = []  # type: List[int]

            if byte == header:
                #
                # Capture the full frame
                #
                while True:
                    byte = (yield)
                    if byte == footer:
                        target.send(frame)
                        break
                    elif byte == dle:
                        byte = (yield)
                        frame.append(after_dle_func(byte))
                    else:
                        frame.append(byte)


    def frame_receiver() -&gt; Generator:
        &quot;&quot;&quot; A simple co-routine &quot;sink&quot; for receiving full frames.&quot;&quot;&quot;
        while True:
            frame = (yield)
            print(&#39;Got frame:&#39;, &#39;&#39;.join(&#39;%02x&#39; % x for x in frame))

    bytestream = bytes(
        bytearray((0x70, 0x24,
                   0x61, 0x99, 0xAF, 0xD1, 0x62,
                   0x56, 0x62,
                   0x61, 0xAB, 0xAB, 0x14, 0x62,
                   0x7)))

    frame_consumer = frame_receiver()
    next(frame_consumer)  # Get to the yield

    unwrapper = unwrap_protocol(target=frame_consumer)
    next(unwrapper)  # Get to the yield

    for byte in bytestream:
        unwrapper.send(byte)

This runs properly...

    $ ./decoder.py 
    Got frame: 99afd1
    Got frame: ab14

...and also typechecks:

    $ mypy --disallow-untyped-defs decoder.py 
    $

But I am pretty sure I can do better than just use the `Generator` base class in the type specs (just as I did for the `Callable`). I know it takes 3 type parameters (`Generator[A,B,C]`), but I am not sure how exactly they&#39;d be specified here.

Any help most welcome.


Proper type annotation of Python functions with yield

How can I find out if a function or method is a normal function or an async function? I would like my code to automatically support normal or async callbacks and need a way to test what type of function is passed.

    async def exampleAsyncCb():
        pass

    def exampleNomralCb():
        pass

    def isAsync(someFunc):
        #do cool dynamic python stuff on the function
        return True/False

	async def callCallback(cb, arg):
		if isAsync(cb):
			await cb(arg)
		else:
			cb(arg)

And depending on what type of function gets passed it should either run it normally or with await. I tried various things but have no idea how to implement `isAsync()`.

Test if function or method is normal or asynchronous

I am used to working with `AsyncTask` and understand it pretty well due to its simplicity. But `Coroutines` are confusing to me. Can you please explain to me in a simple way what is the difference and purpose of each of the following?

1. `GlobalScope.launch(Dispatchers.IO) {}`
2. `GlobalScope.launch{}`
3. `CoroutineScope(Dispatchers.IO).launch{}`
4. `lifecycleScope.launch(Dispatchers.IO){}`
5. `lifecycleScope.launch{}`

GlobalScope vs CoroutineScope vs lifecycleScope

I&#39;ve read that in CPython, the interpreter stack (the list of Python functions called to reach this point) is mixed with the C stack (the list of C functions that were called in the interpreter&#39;s own code). If so, then how are generators and coroutines implemented? How do they remember their execution state? Does CPython copy each generator&#39;s / coroutine&#39;s stack to and from an OS stack? Or does CPython simply keep the generator&#39;s topmost stack frame on the heap, since the generator can only yield from that topmost frame?

All Coroutine Solutions on Gang of Coders

Total of 27 Coroutine Solutions

What does the "yield from" syntax do in asyncio and how is it different from "await"

Suspend function 'callGetApi' should be called only from a coroutine or another suspend function