path: root/python/atms-310/notebooks/Week 05 M.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>9. Data Analysis Exercise</h1>\n",
    "<h2>10/23/2023</h2>\n",
    "\n",
    "<h2>9.0 Last Time...</h2>\n",
    "<ul>\n",
    "    <li>The <b>open()</b> statement lets you open a file in read, write, or append mode.</li>\n",
    "    <li>Files should always be closed using the <b>close()</b> statement.</li>\n",
    "    <li>You can read a single line with <b>readline()</b>, and multiple lines with <b>readlines()</b>.</li>\n",
    "    <li>The <b>write()</b> method allows you to write a single line, and the <b>writelines()</b> method allows you to write multiple lines.</li>\n",
    "    <li><b>split()</b> lets you break strings based on defined separators.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>9.1 The General Idea...</h2>\n",
    "\n",
    "Today we're going to be working our way through various ways of analyzing datasets. The datasets in question are called <b>data0001.txt</b>, <b>data0002.txt</b>, and <b>data0003.txt</b>.\n",
    "\n",
    "These datasets are just composed of randomly generated numbers that have particular statistical features. We're going to make use of file I/O techniques to calculate certain statistics for each of them.\n",
    "\n",
    "<h3>9.1.1 A Quick Review of Statistics</h3>\n",
    "\n",
    "As a review, the <b>mean</b> is what we typically think of as \"average\": the sum of all elements, divided by the total number of elements. A mean is a useful summary, but it is <b>sensitive to outliers</b>. As an example, consider the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "a = np.array([1,2,3,4,5])\n",
    "print(np.mean(a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "31113.0\n"
     ]
    }
   ],
   "source": [
    "# Now consider an outlier: a value that's much higher or lower than the others.\n",
    "a = np.array([1,2,3,4,155555])\n",
    "print(np.mean(a))\n",
    "# the mean is not resistant to outliers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we suspect there are outliers in the data, we can instead use the <b>median</b>, which is not sensitive to outliers. The median simply organizes all the values in ascending order and picks the middle one (or averages the middle two). The median is said to be <b>resistant to outliers</b>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.0\n"
     ]
    }
   ],
   "source": [
    "a = np.array([1,2,3,4,55555])\n",
    "print(np.median(a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding the mean or median of a dataset is only part of the story: we're often also interested in how spread out the data are. Consider the following examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22221.000022501237\n"
     ]
    }
   ],
   "source": [
    "print(np.std(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The arrays have the same mean (and the same median, actually), but the spread of values is very different. As a result, we use measures of spread such as the <b>standard deviation</b>, which is essentially a measure of the average distance between each value and the mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can tell that the second array is more spread out than the first.\n",
    "\n",
    "Okay, but how does the standard deviation do when it comes to outliers?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a big difference there! So just like how the median can be used as a statistic instead of the mean when we suspect there are outliers, we can also use an outlier-resistant measure of spread called the <b>inter-quartile range (IQR)</b>.\n",
    "\n",
    "After sorting the data in ascending order, a <b>quartile</b> corresponds to a quarter of the data. Counting upward through the data, once we've reached 1/4 of the data, we've reached the first quartile. The second quartile is when we've reached 1/2 of the data (so the <b>second quartile is equal to the median</b>). The third quartile is when we've reached 3/4 of the data.\n",
    "\n",
    "The interquartile range is simply the difference between the 3rd quartile and the 1st quartile. This function isn't in NumPy (yet!), but it is in scipy.stats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.0\n",
      "3.0\n"
     ]
    }
   ],
   "source": [
    "import scipy.stats as S\n",
    "import numpy as np\n",
    "\n",
    "a = np.array([1,2,3,4,5,6,7])\n",
    "b = np.array([1,2,3,4,5,6,1552345])\n",
    "\n",
    "print(S.iqr(a))\n",
    "print(S.iqr(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to get fancy with your analysis, there's also <b>skewness</b> and <b>kurtosis</b>, but they're harder to puzzle out by hand.\n",
    "\n",
    "<b>Skewness</b> is a measure of how asymmetrical your distribution is: negative skew means a plot of the data has a longer left tail, whereas positive skew means a plot of the data has a longer right tail.\n",
    "\n",
    "<b>Kurtosis</b> is a measure of how sharp the peak is in a distribution, as compared to a Gaussian (bell-curve). If the kurtosis is greater than 3, it's got a sharper curve than a Gaussian distribution. If it's less than 3, it's got a more gradual curve than a Gaussian distribution.\n",
    "\n",
    "These are more complicated statistics, but you may come across the names, and they can come in handy when you're doing data analysis!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n",
      "2.0412414522829976\n",
      "-1.25\n",
      "2.1666666665875907\n"
     ]
    }
   ],
   "source": [
    "print(S.skew(a))\n",
    "print(S.skew(b))\n",
    "\n",
    "print(S.kurtosis(a))\n",
    "print(S.kurtosis(b))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>9.2 A Traditional Approach</h2>\n",
    "\n",
    "We're going to call this a traditional approach because it's the sort of thing you could do in just about any programming language; it doesn't really take advantage of the power of Python.\n",
    "\n",
    "Our goal is to calculate the mean, median, standard deviation, IQR, skewness, and kurtosis of each of the 3 datasets!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start by importing the relevant packages.\n",
    "import numpy as np\n",
    "import scipy.stats as s\n",
    "\n",
    "# Let's create a function that will read data from any file.\n",
    "# The function has one argument: the name of the file.\n",
    "def read(file):\n",
    "    fileobj = open(file, \"r\")\n",
    "    # Start by defining a file object. We're opening in read-only mode.\n",
    "    outputstr = fileobj.readlines()\n",
    "    # Next, use readlines() to create a variable containing all the data.\n",
    "    fileobj.close()\n",
    "    # Close the file!\n",
    "    outputarray = np.zeros(len(outputstr))\n",
    "    # Let's initalize an array that will contain all the individual values from the file.\n",
    "    for n in np.arange(len(outputstr)):\n",
    "        outputarray[i] = float(outputstr[i]) \n",
    "    # Finally, let's loop over all the lines and put their values into this new array.\n",
    "    return outputarray\n",
    "    # We now have a function that takes in a file name and puts\n",
    "    # all the data into an array!\n",
    "    # The final step is to return the data array.\n",
    "\n",
    "\n",
    "# Okay, so let's make use of this function for our three datasets.\n",
    "data1 = read(\"../datasets/data001.txt\")\n",
    "data2 = read(\"../datasets/data002.txt\")\n",
    "data3 = read(\"../datasets/data003.txt\")\n",
    "\n",
    "# Calculate the stats!\n",
    "mean1 = np.mean(data1)\n",
    "mean1 = np.mean(data1)\n",
    "mean1 = np.mean(data1)\n",
    "\n",
    "\n",
    "# Printing:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>9.3 Array Storage</h2>\n",
    "\n",
    "We can do better than that! Let's make use of arrays for the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the necessary packages.\n",
    "import numpy as np\n",
    "import scipy.stats as s\n",
    "\n",
    "# Let's initialize arrays of our final values!\n",
    "numfiles = 3\n",
    "mean = np.zeros(numfiles)\n",
    "median = np.zeros(numfiles)\n",
    "std = np.zeros(numfiles)\n",
    "\n",
    "# Now, let's use a loop to calculate the values!\n",
    "    # We can use the index from the loop to name each file!\n",
    "\n",
    "    # Now, just use readdata() to grab all the data from the file.\n",
    "\n",
    "    # Calculate your statistics!\n",
    "\n",
    "\n",
    "# Once the loop is complete, print out the arrays.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That worked fairly well, but the big concern here is that sometimes your files won't be as nicely numbered as they are.\n",
    "\n",
    "<h2>9.4 Dictionary Storage</h2>\n",
    "\n",
    "How can we use dictionaries to our advantage? This might solve our problem with our filenames! Instead of relying on them to be a perfectly numbered list, we can use them as keys in a dictionary.\n",
    "\n",
    "And there's a new import command we can use that will grab all the file names!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The usual suspects.\n",
    "import numpy as np\n",
    "import scipy.stats as s\n",
    "# And a new friend!\n",
    "import glob\n",
    "\n",
    "# Let's start by getting a list of files in the directory.\n",
    "# We don't want to grab EVERYTHING, so we'll say it has to start with the word 'data' and end with '.txt.'\n",
    "filelist = glob.glob(\"../datasets/data*.txt\")\n",
    "filelist.sort()\n",
    "\n",
    "# Now initialize our dictionaries as empty to begin with.\n",
    "mean = {}\n",
    "median = {}\n",
    "stddev = {}\n",
    "iqr = {}\n",
    "skewness = {}\n",
    "kurtosis = {}\n",
    "\n",
    "# Loop through all files.\n",
    "for i in filelist:\n",
    "    # Read the data.\n",
    "    data = read(i)\n",
    "    # Assign key-value pairs!\n",
    " \n",
    "    \n",
    "# And, outside the loop, print the results.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>9.5 MORE Dictionary Storage</h2>\n",
    "\n",
    "Okay, well, what if we didn't want to have to make a separate dictionary for every statistical metric? Remember, the key:value pairs in dictionaries are very flexible and actually allow you to put dictionaries themselves into the values!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Old friends, back again.\n",
    "\n",
    "\n",
    "# First, create a dictionary of metrics with the commands you'll need to calculate them.\n",
    "\n",
    "# And we get our files the usual way.\n",
    "\n",
    "# Now let's initialize a results dictionary for each metric.\n",
    "\n",
    "# Now loop through all files, storing the relevant metrics!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The power of what Python's enabled us to do here is that we can change almost anything very easily: adding or removing files, adding or removing metrics, it's all done with one or two lines of code at most. The first version we saw would have been <b>much</b> more complicated!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>9.6 Take-Home Points</h2>\n",
    "<ul>\n",
    "    <li>Statistics that are not resistant to outliers include the mean, the standard deviation, skewness, and kurtosis.</li>\n",
    "    <li>Statistics that are resistant to outliers include the median and the interquartile range.</li>\n",
    "    <li>By making use of dictionaries, we can create versatile, non-hard-coded programs!</li>\n",
    "    <li>glob is a package that enables us to grab all files within a directory</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}