I want to separate my data into train and test set, should I apply normalization over data before or after the split? Does it make any difference while building predictive model? 


Normalize data before or after split of training and testing data?

I want to write a calendar application. It is really recurring items that throw a wrench in the works for the DB schema. I would love some input on how to organize this.

What if a user creates an event, and inputs that it repeats everyone Monday, forever? How could I store all that in the database? I can&#39;t create infinite events. Do I simply put a table in there that holds the relevant info so I can calculate where all the events go? If so, I would have to calculate them every time the user views a new part of the calendar. What if they page through the months, but they have a ton of recurring items?

Also, the schema needs to handle when a user clicks an item and says &quot;Edit this one in the sequence&quot; not all items in the sequence. Do I then split the one item off of the sequence?

**Update 1**

I have not looked at iCal at all. To be clear, I think saving the info that allows you to calculate the recurring items, and splitting off any that differ from the sequence is a great way to store it to be able to transfer it. But I think that in an application, this would be too slow, to do the date math all over the place.

Laying out a database schema for a calendar application

This has become quite a frustrating question, but I&#39;ve asked in the Coursera discussions and they won&#39;t help. Below is the question:

![enter image description here][1]

I&#39;ve gotten it wrong 6 times now. How do I normalize the feature? Hints are all I&#39;m asking for.

I&#39;m assuming x_2^(2) is the value 5184, unless I am adding the x_0 column of 1&#39;s, which they don&#39;t mention but he certainly mentions in the lectures when talking about creating the design matrix X. In which case x_2^(2) would be the value 72. Assuming one or the other is right (I&#39;m playing a guessing game), what should I use to normalize it? He talks about 3 different ways to normalize in the lectures: one using the maximum value, another with the range/difference between max and mins, and another the standard deviation -- they want an answer correct to the *hundredths*. Which one am I to use? This is so confusing.

  [1]: http://i.stack.imgur.com/OK0L3.png

Normalize a feature in this table

I have sales people and bean counters who are trying to sell customizations to clients, which is fine.  But when a complex change request comes in that I send back a large estimate for, they get confused.  Often they come back at me with &quot;Why can&#39;t you just add another column?&quot; which by another, they mean a dozen or so custom columns PER client.

So far all I can come back with is &quot;We are trying to keep the database well normalized&quot; which means nothing to them.  I tell them I can create a system of tables that allows each client to define their own set of custom fields, but of course that takes more time and money than &quot;just adding a few columns&quot;.  And of course they want to have their cake and eat it too.

So how can I make them understand?

Explaining why &quot;Just add another column to the DB&quot; is a bad idea, to non programmers

I tried to run the graph cut algorithm for a slice of an MRI after converting it into PNG format. I keep encountering the following problem:
```none
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
```  


This is even after setting `vmin` and `vmax` as follows:

    plt.imshow(out, vmin=0, vmax=255)

Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers)

I need to compare 2 strings as equal such as these:

&gt;Lubeck == L&#252;beck

In JavaScript.

Why? Well, I have an auto-completion field that&#39;s going out to a Java service using Lucene, where place names are stored naturally (as L&#252;beck), but also indexed as normalized text, 

    import sun.text.Normalizer;
    oDoc.setNameLC = Normalizer.normalize(oLocName, Normalizer.DECOMP, 0)
        .toLowerCase().replaceAll(&quot;[^\\p{ASCII}]&quot;,&quot;&quot;);

This way some-one who doesn&#39;t know to type &quot;M&#232;xico&quot; can type &quot;mexico&quot; and get a match which returns &quot;M&#232;xico&quot; (among a lot of other possible hits, like &quot;Caf&#233; M&#232;xico, Dubai, UAE&quot;).

Now the thing is I don&#39;t have the ability to change the service to do any highlighting on the server side, therefore I am highlighting on the client JavaScript side with something like:

    return result.replace( input.replace(/[aeiou]/g,&quot;.&quot;), &quot;&lt;b&gt;$1&lt;/b&gt;&quot;);

It&#39;s a little more fancy because I am escaping special regex characters in the input. This is fine for simple one word matches at the beginning of a hit, but it really breaks down if you suddenly wish to support multi-word matches like &quot;london cafe&quot;:

    input = input.strip().toLowerCase(); //fyi prototype&#39;s strip is like trim
    re = new RegEx(input.replace(/[aeiou]/g,&quot;.&quot;).replace(/\s+/g,&quot;|&quot;),&quot;gi&quot;);
    return result.replace(re, &quot;&lt;b&gt;$1&lt;/b&gt;&quot;);

This doesn&#39;t work for say &quot;london ca&quot; (was typing london cafe), because it would mark &quot;Jack London Cabin, Dawson City, Canada&quot; as: `&quot;Ja&lt;b&gt;ck&lt;/b&gt; &lt;b&gt;London&lt;/b&gt; &lt;b&gt;ca&lt;/b&gt;bin, Dawson &lt;b&gt;Ci&lt;/b&gt;ty, &lt;b&gt;Ca&lt;b/&gt;nada&quot;` [note the &quot;ck&quot; and &quot;Ci&quot; particularly]

Therefore I&#39;m sort of looking for something that&#39;s not as crazy as:

    input = input.strip().toLowerCase();
    input = input.replace(/a/g,&quot;[&#192;&#224;&#193;&#225;&#194;&#226;&#195;&#227;&#196;&#228;&#197;&#229;&#198;&#230;ĀāĂăĄą]&quot;);
    input = input.replace(/e/g,&quot;[&#200;&#232;&#201;&#233;&#202;&#234;&#203;&#235;ĒēĔĕĖėĘęĚě]&quot;);
    // ditto for i, o, u, y, c, n, maybe also d, g, h, j, k, l, r, s, t, w, z 
    re = new RegEx(input.replace(/\s+/g,&quot;|&quot;),&quot;gi&quot;);
    return result.replace(re, &quot;&lt;b&gt;$1&lt;/b&gt;&quot;);

Is there a compiled table I can refer to mapping a range of characters which are accented versions of an other character to that character, by which I don&#39;t mean the plain unicode chart. And if so, could I avoid using weird, possibly slow, RegEx statements?

*About the bounty:*  
Before I started a bounty there were two answers, the one pointing me to doing it in Ruby, and [the one][1] that [MizzardX][2] wrote which was a completion of the basic form I&#39;d put in my question. Now don&#39;t get me wrong, I really appreciate working it out as completely as he did, but I just wished that there might be another way.  It seems so far that everyone who&#39;s dropped by to look at the question and answer has decided that MizzardX covers it just fine, or that they have no different approach. I would be interested in a different approach, and if it simply isn&#39;t available before the bounty closes, MizzardX will win the bounty (though in a cruel twist, his edits mad it a community wiki answer, so I&#39;m not sure if he&#39;ll get the bounty!)

    


  [1]: https://stackoverflow.com/questions/227950/programatic-accent-reduction-in-javascript-aka-text-normalization-or-unaccenting/228006#228006
  [2]: https://stackoverflow.com/users/22364/mizardx

Programatic Accent Reduction in JavaScript (aka text normalization or unaccenting)

I would like to convert a NumPy array to a unit vector. More specifically, I am looking for an equivalent version of this normalisation function:

    def normalize(v):
        norm = np.linalg.norm(v)
        if norm == 0: 
           return v
        return v / norm

This function handles the situation where vector `v` has the norm value of 0.

Is there any similar functions provided in `sklearn` or `numpy`?

How to normalize a NumPy array to a unit vector?

I have a dataset called `spam` which contains 58 columns and approximately 3500 rows of data related to spam messages. 

I plan on running some linear regression on this dataset in the future, but I&#39;d like to do some pre-processing beforehand and standardize the columns to have zero mean and unit variance. 

I&#39;ve been told the best way to go about this is with R, so I&#39;d like to ask **how can i achieve normalization with R**? I&#39;ve already got the data properly loaded and I&#39;m just looking for some packages or methods to perform this task.  

Standardize data columns in R



Why do we have to normalize the input for a neural network?

I understand that sometimes, when for example the input values are non-numerical a certain transformation must be performed, but when we have a numerical input? Why the numbers must be in a certain interval?

What will happen if the data is not normalized?

Why do we have to normalize the input for an artificial neural network?

Given a 3 times 3 numpy array

    a = numpy.arange(0,27,3).reshape(3,3)

    # array([[ 0,  3,  6],
    #        [ 9, 12, 15],
    #        [18, 21, 24]])


To normalize the rows of the 2-dimensional array I thought of

    row_sums = a.sum(axis=1) # array([ 9, 36, 63])
    new_matrix = numpy.zeros((3,3))
    for i, (row, row_sum) in enumerate(zip(a, row_sums)):
        new_matrix[i,:] = row / row_sum

There must be a better way, isn&#39;t there?

Perhaps to clearify: By normalizing I mean, the sum of the entrys per row must be one. But I think that will be clear to most people.

How to normalize a 2-dimensional numpy array in python less verbose?

When using `SQL` or `MySQL` (or any relational DB for that matter) - I understand that saving the data in regular columns is better for indexing sake and other purposes...

The thing is loading and saving `JSON` data is sometimes a lot more simple - and makes the development easier.

Are there any &quot;golden rules&quot; for saving raw `JSON` data in the DB?

Is it absolutely a bad practice to do so?

When can I save JSON or XML data in an SQL Table

I&#39;d like to know do I normalize a URL in python. 

For example, If I have a url string like : &quot;http://www.example.com/foo goo/bar.html&quot;

I need a library in python that will transform the extra space (or any other non normalized character) to a proper URL.

How can I normalize a URL in python

I heard a lot about denormalization which was made to improve performance of certain application. But I&#39;ve never tried to do anything related.

So, I&#39;m just curious, which places in normalized DB makes performance worse or in other words, what are denormalization principles?

How can I use this technique if I need to improve performance?   

In what way does denormalization improve database performance?

Before I dive really deep into MongoDB for days, I thought I&#39;d ask a pretty basic question as to whether I should dive into it at all or not. I have basically no experience with nosql.

I did read a little about some of the benefits of document databases, and I think for this new application, they will be really great. It is always a hassle to do favourites, comments, etc. for many types of objects (lots of m-to-m relationships) and subclasses - it&#39;s kind of a pain to deal with.

I also have a structure that will be a pain to define in SQL because it&#39;s extremely nested and translates to a document a lot better than 15 different tables.

But I am confused about a few things. 

1. Is it desirable to keep your database normalized still? I really don&#39;t want to be updating multiple records. Is that still how people approach the design of the database in MongoDB?

2. What happens when a user favourites a book and this selection is still stored in a user document, but then the book is deleted? How does the relationship get detached without foreign keys? Am I manually responsible for deleting all of the links myself?

3. What happens if a user favourited a book that no longer exists and I query it (some kind of join)? Do I have to do any fault-tolerance here?

MongoDB normalization, foreign key and joining

I&#39;m using the `MinMaxScaler` model in sklearn to normalize the features of a model.

    training_set = np.random.rand(4,4)*10
    training_set

           [[ 6.01144787,  0.59753007,  2.0014852 ,  3.45433657],
           [ 6.03041646,  5.15589559,  6.64992437,  2.63440202],
           [ 2.27733136,  9.29927394,  0.03718093,  7.7679183 ],
           [ 9.86934288,  7.59003904,  6.02363739,  2.78294206]]


    scaler = MinMaxScaler()
    scaler.fit(training_set)    
    scaler.transform(training_set)


       [[ 0.49184811,  0.        ,  0.29704831,  0.15972182],
       [ 0.4943466 ,  0.52384506,  1.        ,  0.        ],
       [ 0.        ,  1.        ,  0.        ,  1.        ],
       [ 1.        ,  0.80357559,  0.9052909 ,  0.02893534]]



Now I want to use the same scaler to normalize the test set:

       [[ 8.31263467,  7.99782295,  0.02031658,  9.43249727],
       [ 1.03761228,  9.53173021,  5.99539478,  4.81456067],
       [ 0.19715961,  5.97702519,  0.53347403,  5.58747666],
       [ 9.67505429,  2.76225253,  7.39944931,  8.46746594]]

But I don&#39;t want so use the `scaler.fit()` with the training data all the time. Is there a way to save the scaler and load it later from a different file?    

All Normalization Solutions on Gang of Coders

Total of 15 Normalization Solutions