If you will execute `model.fit(X_train, y_train)` for a second time - it&#39;ll overwrite all previously fitted coefficients, weights, intercept (bias), etc.

If you want to fit just a portion of your data set and then to improve your model by fitting a new data, then you can use [estimators, supporting &quot;Incremental learning&quot; (those, that implement `partial_fit()` method)](https://scikit-learn.org/0.15/modules/scaling_strategies.html)

You can use term **fit()** and **train()** word interchangeably in machine learning. Based on classification model you have instantiated, may be a `clf = GBNaiveBayes()` or `clf = SVC()`,  your model uses specified machine learning technique.&lt;br&gt;
And as soon as you call `clf.fit(features_train, label_train)` your model starts training using the features and labels that you have passed.

you can use `clf.predict(features_test)` to predict.&lt;br&gt;
If you will again call `clf.fit(features_train2, label_train2)` it will start training again using passed data and will remove the previous results. Your model will **reset** the following inside model:

 - Weights
 - Fitted Coefficients
 - Bias
 - And other training related stuff...

You can use **partial_fit()** method as well if you want your previous calculated stuff to stay and additionally train using next data



I&#39;m using a transparent AppBar and I would like to display body behind the transparent AppBar and not bellow.
How to do that ?

How to display body under the AppBar of scaffold widget and not below?

I am developing a web application with a REST API using C# with ASP.NET Core 2.0.

What I want to achieve is when the client send a request to an endpoint I will run a background task separated from the client request context which will be ended if the task started successfully.

I know there is `HostedService` but the problem is that the `HostedService` starts when the server starts, and as far as I know there is no way to start the `HostedService` manually from a controller.

Here is a simple code that demonstrates the question.
```
[Authorize(AuthenticationSchemes = &quot;UsersScheme&quot;)]
public class UsersController : Controller
{
    [HttpPost]
    public async Task&lt;JsonResult&gt; StartJob([FromForm] string UserId, [FromServices] IBackgroundJobService backgroundService)
    {
        // check user account
        (bool isStarted, string data) result = backgroundService.Start();

        return JsonResult(result);
    }
}
```

Run a background task from a controller action in ASP.NET Core

After I instantiate a scikit model (e.g. `LinearRegression`), if I call its `fit()` method multiple times (with different `X` and `y` data), what happens? Does it fit the model on the data like if I just re-instantiated the model (i.e. from scratch), or does it keep into accounts data already fitted from the previous call to `fit()`?

Trying with `LinearRegression` (also looking at its source code) it seems to me that every time I call `fit()`, it fits from scratch, ignoring the result of any previous call to the same method. I wonder if this true in general, and I can rely on this behavior for all models/pipelines of scikit learn.

What does calling fit() multiple times on the same model do?

<p>After I instantiate a scikit model (e.g. <code>LinearRegression</code>), if I call its <code>fit()</code> method multiple times (with different <code>X</code> and <code>y</code> data), what happens? Does it fit the model on the data like if I just re-instantiated the model (i.e. from scratch), or does it keep into accounts data already fitted from the previous call to <code>fit()</code>?</p>
<p>Trying with <code>LinearRegression</code> (also looking at its source code) it seems to me that every time I call <code>fit()</code>, it fits from scratch, ignoring the result of any previous call to the same method. I wonder if this true in general, and I can rely on this behavior for all models/pipelines of scikit learn.</p>


**Especially** when using recursive code there are massive improvements with `lru_cache`.       I do understand that a cache is a space that stores data that has to be served fast and saves the computer from recomputing.

How does the **Python** `lru_cache` from functools work internally? 

I&#39;m Looking for a specific answer, does it use dictionaries like the rest of Python? Does it only store the `return` value?

I know that **Python** is heavily built on top of dictionaries, however, I couldn&#39;t find a specific answer to this question. Hopefully, someone can simplify this answer for all the users on ***StackOverflow***.



How does Lru_cache (from functools) Work?

I&#39;m looking for a way to hide code cells (inputs) when export my .iipynb file to a HTML. I don&#39;t want the code cells to be visible at all (not some button that turn them off/on). The output is for people that have no idea what a programming language is. I tried many things that I found on the internet but nothing seems to work. 

Thanks

Hide Code when exporting Jupyter notebook to HTML

I have two dataframes in python. I want to update rows in first dataframe using  matching values from another dataframe. Second dataframe serves as an override. 

Here is an example with same data and code:  

DataFrame 1 : 

[![enter image description here][1]][1]

DataFrame 2: 

[![enter image description here][2]][2]

I want to update update dataframe 1 based on matching code and name. In this example Dataframe 1 should be updated as below: 

[![enter image description here][3]][3]

Note : Row with Code =2 and Name= Company2 is updated with value 1000 (coming from Dataframe 2) 

    import pandas as pd
    
    data1 = {
             &#39;Code&#39;: [1, 2, 3],
             &#39;Name&#39;: [&#39;Company1&#39;, &#39;Company2&#39;, &#39;Company3&#39;],
             &#39;Value&#39;: [200, 300, 400],
             
        }
    df1 = pd.DataFrame(data1, columns= [&#39;Code&#39;,&#39;Name&#39;,&#39;Value&#39;])
    
    data2 = {
             &#39;Code&#39;: [2],
             &#39;Name&#39;: [&#39;Company2&#39;],
             &#39;Value&#39;: [1000],
        }
    
    df2 = pd.DataFrame(data2, columns= [&#39;Code&#39;,&#39;Name&#39;,&#39;Value&#39;])

Any pointers or hints? 

  [1]: https://i.stack.imgur.com/QPJs2.png
  [2]: https://i.stack.imgur.com/Xbpmh.png
  [3]: https://i.stack.imgur.com/JA3Fl.png

Python Pandas update a dataframe value from another dataframe

I am dealing with a legacy Dockerfile. Here is a _very simplified_ version of what I am dealing with:

    FROM ubuntu:14.04
    
    RUN apt-get -y update &amp;&amp; apt-get -y install \
        python-pip \
        python-numpy # ...and many other packages
    
    RUN pip install -U pip
    
    RUN pip install -r /tmp/requirements1.txt # includes e.g., numpy==1.13.0
    RUN pip install -r /tmp/requirements2.txt
    RUN pip install -r /tmp/requirements3.txt

First, several packages are installed using `apt`, and then several packages are installed using `pip`. `pip` version 10 has been released, and [part of the release](https://pip.pypa.io/en/stable/news/#deprecations-and-removals) is this new restriction:

&gt; Removed support for uninstalling projects which have been installed using distutils. distutils installed projects do not include metadata indicating what files belong to that install and thus it is impossible to actually uninstall them rather than just remove the metadata saying they&#39;ve been installed while leaving all of the actual files behind. 

This leads to the following problem in my setup. For example, first `apt` installs `python-numpy`. Later `pip` tries to install a newer version of `numpy` from e.g., `/tmp/requirements1.txt`, and tries to uninstall the older version, but because of the new restriction, it cannot remove this version:

    Installing collected packages: numpy
      Found existing installation: numpy 1.8.2
    Cannot uninstall &#39;numpy&#39;. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.


Now I know at this point there are several solutions.

I could not install `python-numpy` through `apt`. However, this causes issues because `python-numpy` installs a few different packages as requirements, and I do not know if another part of the system relies on these packages. And in reality, there are several `apt` packages installed through the Dockerfile, and each one I remove seems to reveal another `Cannot uninstall X` error, and removes a number of other packages along with it, that our app may or may not rely on. 

I could also use the `--ignore-installed` option when I try to `pip` install things that have already been installed through `apt`, but then again I have the same problem of every `--ignore-installed` argument revealing yet another thing that needs to be ignored.

I could pin `pip` at an older version that does not have this restriction, but I don&#39;t want to be stuck using an outdated version of `pip` forever. 

I have been going around in circles trying to come up with a good solution that involves minimal changes to this legacy Dockerfile, and allows the app we deploy with that file to continue to function as it has been. Any suggestions as to how I can safely get around this problem of `pip` 10 not being able to install newer versions of `distutils` packages? Thank you!

# UPDATE:

I did not realize that `--ignore-installed` could be used without a package as an argument to ignore all installed packages. I am considering whether or not this might be a good option for me, and have asked about it [here][1].


  [1]: https://stackoverflow.com/q/49932926/3642398

pip 10 and apt: how to avoid &quot;Cannot uninstall X&quot; errors for distutils packages

I have a problem when I try to use pip in any way. I&#39;m using Ubuntu 16.04.4

I should say that I&#39;ve used it already, and I never had any problem, but starting today when I use any command I always get the same error (as an example using `pip --upgrade`).

    Traceback (most recent call last):
      File &quot;/usr/local/bin/pip&quot;, line 7, in &lt;module&gt;
        from pip._internal import main
    ImportError: No module named _internal

I have tried doing `sudo apt-get remove python-pip` followed by `sudo apt-get install python-pip` but nothing changed.

pip: no module named _internal

I have performed a PCA analysis over my original dataset and from the compressed dataset transformed by the PCA I have also selected the number of PC I want to keep (they explain almost the 94% of the variance). Now I am struggling with the identification of the original features that are important in the reduced dataset. 
How do I find out which feature is important and which is not among the remaining Principal Components after the dimension reduction?
Here is my code:


    from sklearn.decomposition import PCA
    pca = PCA(n_components=8)
    pca.fit(scaledDataset)
    projection = pca.transform(scaledDataset)

Furthermore, I tried also to perform a clustering algorithm on the reduced dataset but surprisingly for me, the score is lower than on the original dataset. How is it possible? 

Feature/Variable importance after a PCA analysis

I have a dataset containing grayscale images and I want to train a state-of-the-art CNN on them. I&#39;d very much like to fine-tune a pre-trained model (like the ones [here](https://github.com/tensorflow/models/tree/master/research/slim#Pretrained)).

The problem is that almost all models I can find the weights for have been trained on the ImageNet dataset, which contains RGB images.

I can&#39;t use one of those models because their input layer expects a batch of shape `(batch_size, height, width, 3)` or `(64, 224, 224, 3)` in my case, but my images batches are `(64, 224, 224)`.

Is there any way that I can use one of those models? I&#39;ve thought of dropping the input layer after I&#39;ve loaded the weights and adding my own (like we do for the top layers). Is this approach correct?

How can I use a pre-trained neural network with grayscale images?

When I increase/decrease batch size of the mini-batch used in SGD, should I change learning rate? If so, then how?

For reference, I was discussing with someone, and it was said that, when batch size is increased, the learning rate should be decreased by some extent. 

My understanding is when I increase batch size, computed average gradient will be less noisy and so I either keep same learning rate or increase it. 

Also, if I use an adaptive learning rate optimizer, like Adam or RMSProp, then I guess I can leave learning rate untouched.

Please correct me if I am mistaken and give any insight on this.

How should the learning rate change as the batch size change?

How do I calculate the output size in a convolution layer?

For example, I have a 2D convolution layer that takes a 3x128x128 input and has 40 filters of size 5x5.

Calculate the output size in convolution layer

After Training, I saved Both Keras whole Model and Only Weights using 

    model.save_weights(MODEL_WEIGHTS) and model.save(MODEL_NAME)


Models and Weights were saved successfully and there was no error.
I can successfully load the weights simply using model.load_weights and they are good to go, but when i try to load the save model via load_model, i am getting an error.


    File &quot;C:/Users/Rizwan/model_testing/model_performance.py&quot;, line 46, in &lt;module&gt;
    Model2 = load_model(&#39;nasnet_RS2.h5&#39;,custom_objects={&#39;euc_dist_keras&#39;: euc_dist_keras})
    File &quot;C:\Users\Rizwan\AppData\Roaming\Python\Python36\site-packages\keras\engine\saving.py&quot;, line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
    File &quot;C:\Users\Rizwan\AppData\Roaming\Python\Python36\site-packages\keras\engine\saving.py&quot;, line 321, in _deserialize_model
    optimizer_weights_group[&#39;weight_names&#39;]]
    File &quot;C:\Users\Rizwan\AppData\Roaming\Python\Python36\site-packages\keras\engine\saving.py&quot;, line 320, in &lt;listcomp&gt;
    n.decode(&#39;utf8&#39;) for n in
    AttributeError: &#39;str&#39; object has no attribute &#39;decode&#39;

I never received this error and i used to load any models successfully. I am using Keras 2.2.4 with tensorflow backend. Python 3.6.
My Code for training is :

    from keras_preprocessing.image import ImageDataGenerator
    from keras import backend as K
    from keras.models import load_model
    from keras.callbacks import ReduceLROnPlateau, TensorBoard, 
    ModelCheckpoint,EarlyStopping
    import pandas as pd

    MODEL_NAME = &quot;nasnet_RS2.h5&quot;
    MODEL_WEIGHTS = &quot;nasnet_RS2_weights.h5&quot;
    def euc_dist_keras(y_true, y_pred):
    return K.sqrt(K.sum(K.square(y_true - y_pred), axis=-1, keepdims=True))
    def main():

    # Here, we initialize the &quot;NASNetMobile&quot; model type and customize the final 
    #feature regressor layer.
    # NASNet is a neural network architecture developed by Google.
    # This architecture is specialized for transfer learning, and was discovered via Neural Architecture Search.
    # NASNetMobile is a smaller version of NASNet.
    model = NASNetMobile()
    model = Model(model.input, Dense(1, activation=&#39;linear&#39;, kernel_initializer=&#39;normal&#39;)(model.layers[-2].output))

    #    model = load_model(&#39;current_best.hdf5&#39;, custom_objects={&#39;euc_dist_keras&#39;: euc_dist_keras})

    # This model will use the &quot;Adam&quot; optimizer.
    model.compile(&quot;adam&quot;, euc_dist_keras)
    lr_callback = ReduceLROnPlateau(monitor=&#39;val_loss&#39;, factor=0.2, patience=5, min_lr=0.003)
    # This callback will log model stats to Tensorboard.
    tb_callback = TensorBoard()
    # This callback will checkpoint the best model at every epoch.
    mc_callback = ModelCheckpoint(filepath=&#39;current_best_mem3.h5&#39;, verbose=1, save_best_only=True)
    es_callback=EarlyStopping(monitor=&#39;val_loss&#39;, min_delta=0, patience=4, verbose=0, mode=&#39;auto&#39;, baseline=None, restore_best_weights=True)

    # This is the train DataSequence.
    # These are the callbacks.
    #callbacks = [lr_callback, tb_callback,mc_callback]
    callbacks = [lr_callback, tb_callback,es_callback]

    train_pd = pd.read_csv(&quot;./train3.txt&quot;, delimiter=&quot; &quot;, names=[&quot;id&quot;, &quot;label&quot;], index_col=None)
    test_pd = pd.read_csv(&quot;./val3.txt&quot;, delimiter=&quot; &quot;, names=[&quot;id&quot;, &quot;label&quot;], index_col=None)

     #    train_pd = pd.read_csv(&quot;./train2.txt&quot;,delimiter=&quot; &quot;,header=None,index_col=None)
     #    test_pd = pd.read_csv(&quot;./val2.txt&quot;,delimiter=&quot; &quot;,header=None,index_col=None)
    #model.summary()
    batch_size=32
    datagen = ImageDataGenerator(rescale=1. / 255)
    train_generator = datagen.flow_from_dataframe(dataframe=train_pd, 
    directory=&quot;./images&quot;, x_col=&quot;id&quot;, y_col=&quot;label&quot;,
                                                  has_ext=True, 
    class_mode=&quot;other&quot;, target_size=(224, 224),
                                                  batch_size=batch_size)
    valid_generator = datagen.flow_from_dataframe(dataframe=test_pd, directory=&quot;./images&quot;, x_col=&quot;id&quot;, y_col=&quot;label&quot;,
                                                  has_ext=True, class_mode=&quot;other&quot;, target_size=(224, 224),
                                                  batch_size=batch_size)

    STEP_SIZE_TRAIN = train_generator.n // train_generator.batch_size
    STEP_SIZE_VALID = valid_generator.n // valid_generator.batch_size
    model.fit_generator(generator=train_generator,
                        steps_per_epoch=STEP_SIZE_TRAIN,
                        validation_data=valid_generator,
                        validation_steps=STEP_SIZE_VALID,
                        callbacks=callbacks,
                        epochs=20)

    # we save the model.
    model.save_weights(MODEL_WEIGHTS)
    model.save(MODEL_NAME)
    if __name__ == &#39;__main__&#39;:
       # freeze_support() here if program needs to be frozen
        main()



Does Any one got &quot;AttributeError: &#39;str&#39; object has no attribute &#39;decode&#39; &quot; , while Loading a Keras Saved Model

What is the difference between the two? It seems that both create new columns, which their number is equal to the number of unique categories in the feature. Then they assign 0 and 1 to data points depending on what category they are in. 

Scikit-learn&#39;s LabelBinarizer vs. OneHotEncoder

I want to get feature names after I fit the pipeline.

    categorical_features = [&#39;brand&#39;, &#39;category_name&#39;, &#39;sub_category&#39;]
    categorical_transformer = Pipeline(steps=[
        (&#39;imputer&#39;, SimpleImputer(strategy=&#39;constant&#39;, fill_value=&#39;missing&#39;)),
        (&#39;onehot&#39;, OneHotEncoder(handle_unknown=&#39;ignore&#39;))])
        
    numeric_features = [&#39;num1&#39;, &#39;num2&#39;, &#39;num3&#39;, &#39;num4&#39;]
    numeric_transformer = Pipeline(steps=[
        (&#39;imputer&#39;, SimpleImputer(strategy=&#39;median&#39;)),
        (&#39;scaler&#39;, StandardScaler())])

    preprocessor = ColumnTransformer(
        transformers=[
            (&#39;num&#39;, numeric_transformer, numeric_features),
            (&#39;cat&#39;, categorical_transformer, categorical_features)])
Then

    clf = Pipeline(steps=[(&#39;preprocessor&#39;, preprocessor),
                          (&#39;regressor&#39;, GradientBoostingRegressor())])

After fitting with pandas dataframe, I can get feature importances from 

`clf.steps[1][1].feature_importances_`

and I tried `clf.steps[0][1].get_feature_names()` but I got an error 

    AttributeError: Transformer num (type Pipeline) does not provide get_feature_names.

How can I get feature names from this?


Sklearn Pipeline: Get feature names after OneHotEncode In ColumnTransformer

I have a dataset consisting of both numeric and categorical data and I want to predict adverse outcomes for patients based on their medical characteristics. I defined a prediction pipeline for my dataset like so:

```python
X = dataset.drop(columns=[&#39;target&#39;])
y = dataset[&#39;target&#39;]

# define categorical and numeric transformers
numeric_transformer = Pipeline(steps=[
    (&#39;knnImputer&#39;, KNNImputer(n_neighbors=2, weights=&quot;uniform&quot;)),
    (&#39;scaler&#39;, StandardScaler())])

categorical_transformer = Pipeline(steps=[
    (&#39;imputer&#39;, SimpleImputer(strategy=&#39;constant&#39;, fill_value=&#39;missing&#39;)),
    (&#39;onehot&#39;, OneHotEncoder(handle_unknown=&#39;ignore&#39;))])

#  dispatch object columns to the categorical_transformer and remaining columns to numerical_transformer
preprocessor = ColumnTransformer(transformers=[
    (&#39;num&#39;, numeric_transformer, selector(dtype_exclude=&quot;object&quot;)),
    (&#39;cat&#39;, categorical_transformer, selector(dtype_include=&quot;object&quot;))
])

# Append classifier to preprocessing pipeline.
# Now we have a full prediction pipeline.
clf = Pipeline(steps=[(&#39;preprocessor&#39;, preprocessor),
                      (&#39;classifier&#39;, LogisticRegression())])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

clf.fit(X_train, y_train)
print(&quot;model score: %.3f&quot; % clf.score(X_test, y_test))
```

However, when running this code, I get the following warning message:
```
ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

    model score: 0.988
```
Can someone explain to me what this warning means? I am new to machine learning so am a little lost as to what I can do to improve the prediction model. As you can see from the numeric_transformer, I scaled the data through standardisation. I am also confused as to how the model score is quite high and whether this is a good or bad thing.





ConvergenceWarning: lbfgs failed to converge (status=1): STOP: TOTAL NO. of ITERATIONS REACHED LIMIT

Importing from pyxdameraulevenshtein gives the following error, I have 

    pyxdameraulevenshtein==1.5.3, 
    pandas==1.1.4 and 
    scikit-learn==0.20.2. 
    Numpy is 1.16.1. 
    Works well in Python3.6, Issue in Python3.7.

 
Has anyone been facing similar issues with Python3.7 (3.7.9), docker image - python:3.7-buster

```    from pyxdameraulevenshtein import normalized_damerau_levenshtein_distance as norm_dl_dist
__init__.pxd:242: in init pyxdameraulevenshtein
    ???
E   ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
```


Content Type	Original Author	Original Content on Stackoverflow
Question	Fanta	View Question on Stackoverflow
Solution 1 - Python	MaxU - stop genocide of UA	View Answer on Stackoverflow
Solution 2 - Python	sgrpwr	View Answer on Stackoverflow

What does calling fit() multiple times on the same model do?

Python Problem Overview

Python Solutions

Solution 1 - Python

Solution 2 - Python

Run a background task from a controller action in ASP.NET Core

How to display body under the AppBar of scaffold widget and not below?

Attributions