Home | Login

Tagged: code

Posts


Loading intraday market data into R Quantmod

10/26/2015

I was excited to find a source for free intraday data last night:

http://freemarketdataapi.barchartondemand.com/

 1 minute resolution, 3 months back for intraday data, 1500 requests per day and 2 years back of daily data, but you can get further back for free from sources like yahoo finance.

I whipped up this little bit of R to chart the API data using quantmod:

load.packages('quantmod')
load.packages('jsonlite')
library(lubridate)

# Here we get 5 min intervals from the API, docs: http://www.barchartondemand.com/api/getHistory
gdxjson <- fromJSON("http://marketdata.websol.barchart.com/getHistory.json?key=<< INSERT YOUR API KEY HERE >>&symbol=GDX&type=minutes&interval=5&startDate=20151019000000")

# Columns that need to be coerced to numeric types, and columns to remove
numeric_columns <- c('open','high','low','close','volume')
rm_columns <- c('timestamp','symbol','tradingDay')

results <- gdxjson$results

# Convert character vector to time objects using lubridate
results$timestamp <- ymd_hms(gdxjson$results$timestamp)
timestamp <- results$timestamp

# Now loop over columns coercing and removing as necessary, is there a better way?
for (col in names(results)) {
  if (is.element(col, numeric_columns)) {
    results[col] <- as.numeric(results[[col]])
  }
  if (is.element(col, rm_columns)) {
    results[col] <- NULL
  }
}

# Convert dataframe to XTS object
gdx_xts <- as.xts(results, timestamp)

chartSeries(gdx_xts)

gist of the code: https://gist.github.com/uberscientist/2330e4a5ba5b05c6bab6

The code is commented, so just read that, and hope it helps any other budget quants out there!

Here's the output chart with the addEMA() run on it.

#r #quantmod #chart #code
9/8/2014

Phoenix Timelapse Tumblr

#! /usr/bin/python

# Script to download pictures and make a gif

from urllib import urlretrieve
from os import system
from time import sleep
import pytumblr

fiveMinuteCounter = 0 
freewayCameraId = 1

threeHourCounter = 1

blogName = "phoenix-timelapse.tumblr.com"

# Authenticate via OAuth
client = pytumblr.TumblrRestClient(
  # Secret API Keys go here
)

homePath = '/home/nak/az511lapse/frames/'

def generateGif():
  path = homePath + str(threeHourCounter) + '/'

  system('convert -coalesce -layers optimize -scale 200 ' + path + '*.jpg ' + path + 'out.gif')
  return

def postPhotoSet():

  # Post photoset of gifs
  postData = []
  postTags = ['cam-' + str(freewayCameraId), 'traffic cam']

  for i in range(1,5):
    gifPath = homePath + str(i) + '/out.gif'
    postData.append(gifPath)
  
  client.create_photo(blogName, tags=postTags, data=postData)
  system('rm frames/1/* frames/2/* frames/3/* frames/4/*')  # delete all the old gifs and frames
  return

def downloadCam():
  global freewayCameraId
  global fiveMinuteCounter
  print "download id:" + str(freewayCameraId)

  # URL to desired camera
  url = 'http://vods.az511.com/adot_' + str(freewayCameraId).zfill(3) + '.jpg'

  path = homePath + str(threeHourCounter) + '/'
  frameNumber = str(fiveMinuteCounter).zfill(2)

  try:
    httpobj = urlretrieve(url, path + frameNumber  + '.jpg');

    if httpobj[1].gettype() == 'text/plain':  #if the camera is down
      freewayCameraId += 1
      downloadCam()

  except:
    print 'Error downloading: ' + url
    system('rm frames/' + str(threeHourCounter) + '/*')
    freewayCameraId += 1
    fiveMinuteCounter = 0
    downloadCam()
    pass

  return


while(1):

  fiveMinuteCounter += 1
  if fiveMinuteCounter >= 37:

    if freewayCameraId > 194:  # switch to the next camera
      freewayCameraId = 1
    else:
      freewayCameraId += 1

    fiveMinuteCounter = 0

    # At this point we've taken 36 pictures
    generateGif()

    threeHourCounter += 1
    if threeHourCounter >= 5:
      # Here we have 4 directories 1 - 4 with a gif for each
      threeHourCounter = 1

      # post to tumblr
      postPhotoSet()

  # Download Camera JPG
  downloadCam()

  # Sleep
  sleep(120)

I should re-write this sometime.  It still gets stuck in random infinite loops sometimes when it can't download a picture or something...

#python #tumblr #timelapse #code