GlobalPrePaidCreditCard.com – Prepaid and gift credit card feature comparison for 9 countries. Sort and filter cards by features to find the right card for your needs.
AWS S3 csv file as D3 report data source
This is an example of how to read a csv file retrieved from an AWS S3 bucket as a data source for a D3 javascript visualization.
The D3 visualization would be an HTML document hosted on a web server.
You will use the AWS SDK to get the csv file from the S3 bucket and so you need to have an AWS S3 bucket key and secret but I won’t cover that in this post.
The key point of this post is to highlight that the bucket.getObject function data is read into D3 using d3.csv.parse(data.Body.toString());
Another note is that d3.csv.parse is for D3 version 3. Older versions use d3.csvParse.
Once implemented, whenever the webpage is refreshed it retrieves latest csv file from the S3 bucket and the D3 visualization is updated.
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.6.3.min.js"></script> <script type="text/javascript"> // aws key and secret (note these should be retrieved from server not put as plain text into html code) AWS.config.accessKeyId = 'xxxxxxxxxxxxxxxxxxxxxxx'; AWS.config.secretAccessKey = 'xxxxxxxxxxxxxxxxxxxxxxx'; AWS.config.region = 'us-east-1'; // create the AWS.Request object var bucket = new AWS.S3(); // use AWS SDK getobject to retrieve csv file bucket.getObject({ Bucket: 'my-S3-bucket', Key: 'myfile.csv' }, // function to use the data retrieve function awsDataFile(error, data) { if (error) { return console.log(error); } // this where magic happens using d3.csv.parse to read myCSVdata.Body.toString() myCSVdata = d3.csv.parse(data.Body.toString()); // now loop through data and get fields desired for visualization var counter = 0; myCSVdata.forEach(function(d) { d.field1= +d.field1; d.field2= +d.field2; countLoop = counter++; }); // now you can create rest of D3 vizualization here // for example like this example https://gist.github.com/d3noob/4414436 my D3 vizualization code here // this closes bucket.getObject }); </script>
Vultr and Digital Ocean make it easy to get projects going fast
Recently I developed a Django web application for a friend. I wanted to be able to get it up and out onto the internet so she could view it and get others to do user testing.
Normally I host my web applications on my Webfaction account. But I wanted to keep this project separate so I went over to Digital Ocean and spun up a virtual server to host the Django web application.
We didn’t have a domain name for this site yet so just used the Digital Ocean provided IP address to access the site and created a Self-Signed SSL Certificate for Apache and forced all traffic to HTTPS to enable secure browsing.
I have used Digital Ocean for development purposes and to share projects with others from time to time. I have also used Vultr.
- Digital Ocean – Use this link to open account and get $10 credit.
- Vultr – use this link to open account and get $20 credit (summer 2016 promotion).
Both have monthly price plans that can also be pro-rated depending on how many days you use them so you can spin them up for a few minutes or days and pay accordingly. I am using the basic plans on both which are good enough for my demonstration development web application purposes.
- Digital Ocean – $5 plan includes 1 cpu, 512MB of RAM, 20GB SSD storage and 1000GB Bandwidth
- Vultr – $5 plan includes 1 cpu, 768MB Memory, 15 GB SSD storage and 1000GB Bandwidth
Beware though that unlike the huge number of shared hosting such as Webfaction, Digital Ocean and Vultr virtual servers are yours to own and manage. You get root access to the server and are responsible for managing the server, OS, how it is configured and secured, etc.
That is definitely something you should consider. There are lots of new things to learn. However there are tutorials and lots of help available on the internet so l recommend at least attempting to setup either or both to learn first hand. The cost is small and risk very low. Knowledge gained and benefits are well worth it.
Many comparison reviews seem to prefer Vultr over Digital Ocean for performance and features. However Digital Ocean was first and has done excellent job of documentation for getting many web applications setup. But Vultr does have good documentation too. You can’t really go wrong with either for development projects. If you don’t like one, just switch to the other!
Django form geocode submitted address to get lat, lon and postal code
One of my Django applications includes a form where user can enter and submit a property address.
The user submitting the form might not know the postal code so I left it optional. However the postal code is a key piece of information for this particular application so I wanted to ensure that I was getting it.
I also wanted to geocode the address immediately to get the address latitude and longitude so it could be shown to user on a Leaflet.js map.
There are lots of free geocoding services for low intensity usage but I ended up using Google Geocoding which is free under certain usage level. You just need to create a Geocoding API project and use the credentials to set up the geocoding.
To interact with the geocoding API I tried Python gecoding modules geopy and geocoder but in the end just used Python Requests module instead as it was less complicated.
When the user clicked the Submit button, behind the scenes, Requests submitted the address to Google’s Geocoding API, gets the JSON response containing the latitude, longitude and postal code which are then written to the application database.
I will update the code in future to check if the user’s postal code is correct and replace it if it is incorrect. Will wait to see how the postal code accuracy looks. Making geocoding API requests too often could bump me over the free usage limit.
The Django View that contains the code described above is shown below.
def property_add(request): property_list = Property.objects.filter(user_id=request.user.id).order_by('created') if request.method == 'POST': form = PropertyForm(request.POST) if form.is_valid(): new_property = form.save(commit=False) address = "%s, %s, %s, %s" % (new_property.address1, new_property.city, new_property.state, new_property.postcode) google_geocode_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxx' url = 'https://maps.googleapis.com/maps/api/geocode/json?address=' + "'" + address + "'" + '&key=' + google_geocode_key try: response = requests.get(url) geoArray = response.json() new_property.lat = geoArray['results'][0]['geometry']['location']['lat'] new_property.lon = geoArray['results'][0]['geometry']['location']['lng'] new_postcode = geoArray['results'][0]['address_components'][7]['long_name'] new_fsa = geoArray['results'][0]['address_components'][7]['short_name'][:3] except: new_property.lat = None new_property.lon = None new_postcode = None new_fsa = None if new_property.postcode: new_property.fsa = new_property.postcode[:3] else: new_property.postcode = new_postcode new_property.fsa = new_fsa new_property.user_id = request.user.id new_property = form.save() return HttpResponseRedirect(reverse(property, args=(new_property.pk,))) else: form = PropertyForm() context_dict = { 'form': form, 'property_list': property_list, } return render( request, 'property_form.html', context_dict, context_instance = RequestContext( request, { 'title':'Add Property', } ) )
Leaflet.js choropleth map color by count using geoJSON datasource
I have a Django web application that needed an interactive map with shapes corresponding to Canadian postal code FSA areas that were different colors based on how many properties were in each FSA. It ended up looking something like the screenshot below.
This exercise turned out to be relatively easy using the awesome open-source Javascript map library Leaflet.js.
I used this Leaflet.js tutorial as the foundation for my map.
One of the biggest challenges was finding a suitable data source for the FSAs. Chad Skelton (now former) data journalist at the Vancouver Sun wrote a helpful blog post about his experience getting a suitable FSA data source. I ended up using his BC FSA data source for my map.
Statistics Canada hosts a Canada Post FSA boundary files for all of Canada. As Chad Skelton notes these have boundaries that extend out into the ocean among other challenges.
Here is a summary of the steps that I followed to get my choropleth map:
1. Find and download FSA boundary file. See above.
2. Convert FSA boundary file to geoJSON from SHP file using qGIS.
3. Create Django queryset to create data source for counts of properties by FSA to be added to the Leaflet map layer.
4. Create Leaflet.js map in HTML page basically the HTML DIV that holds the map and separate Javascript script that loads Leaflet.js, the FSA geoJSON boundary data and processes it to create the desired map.
Find and download FSA boundary file.
See above.
Convert FSA boundary file to geoJSON from SHP file using qGIS.
Go to http://www.qgis.org/en/site/ and download qGIS. Its free and open source.
Use qGIS to convert the data file from Canada Post or other source to geoJSON format. Lots of blog posts and documentation about how to use qGIS for this just a Google search away.
My geoJSON data source looked like this:
var bcData = { "type": "FeatureCollection", "crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:EPSG::4269" } }, "features": [ { "type": "Feature", "properties": { "CFSAUID": "V0A", "PRUID": "59", "PRNAME": "British Columbia \/ Colombie-Britannique" }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ -115.49499542, 50.780018587000029 ], [ -115.50032807, 50.77718343600003 ], [ -115.49722732099997, 50.772528975000057 ], [ -115.49321284, 50.770504059000075 ], [ -115.49393662599999, 50.768143038000062 ], [ -115.50289288699997, 50.762270941000054 ], [ -115.50846411599997, 50.754243300000041 ], [ -115.5104796, 50.753297703000044 ], [ -115.51397592099994, 50.748953800000038 ], [ -115.51861431199995, 50.745737989000077 ], [ -115.52586378899997, 50.743771099000071 ], [ -115.53026371899995, 50.74397910700003 ], [ -115.53451319199996,
Create Django queryset to create data source for counts of properties by FSA to be added to the Leaflet map layer.
I used a SQL query in the Django View to get count of properties by FSA.
This dataset looks like this in the template. These results have only one FSA, if it had more it would have more FSA / count pairs.
Below is code for the Django view query to create the fsa_array FSA / counts data source.
cursor = connection.cursor() cursor.execute( "select fsa, count(*) \ from properties \ group by fsa \ order by fsa;") fsas_cursor = list(cursor.fetchall()) fsas_array = [(x[0].encode('utf8'), int(x[1])) for x in fsas_cursor]
My Javascript largely retains the Leaflet tutorial code with some modifications:
1. How the legend colors and intervals are assigned is changed but otherwise legend functions the same.
2. Significantly changed how the color for each FSA is assigned. The tutorial had the color in its geoJSON file so only had to reference it directly. My colors were coming from the View so I had to change code to include new function to match FSA’s in both my Django view data and the geoJSON FSA boundary file and return the appropriate color based on the Django View data set count.
var fsa_array = [["V3J", 19]]; var map = L.map('map',{scrollWheelZoom:false}).setView([ active_city_center_lat, active_city_center_lon], active_city_zoom); map.once('focus', function() { map.scrollWheelZoom.enable(); }); var fsa_array = fsas_array_safe; L.tileLayer('https://api.tiles.mapbox.com/v4/{id}/{z}/{x}/{y}.png?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpandmbXliNDBjZWd2M2x6bDk3c2ZtOTkifQ._QA7i5Mpkd_m30IGElHziw', { maxZoom: 18, attribution: 'Map data ©' + OpenStreetMap CC-BY-SA' + 'Imagery ©' + 'Mapbox' id: 'mapbox.light' }).addTo(map); // control that shows state info on hover var info = L.control(); info.onAdd = function (map) { this._div = L.DomUtil.create('div', 'info'); this.update(); return this._div; }; info.update = function (props) { this._div.innerHTML = (props ? '' + props.CFSAUID + ' ' + getFSACount(props.CFSAUID) + ' lonely homes' : 'Hover over each postal area to see lonely home counts to date.'); }; info.addTo(map); // get color function getColor(n) { return n > 30 ? '#b10026' : n > 25 ? '#e31a1c' : n > 25 ? '#fc4e2a' : n > 20 ? '#fd8d3c' : n > 15 ? '#feb24c' : n > 10 ? '#fed976' : n > 5 ? '#ffeda0' : n > 0 ? '#ffffcc' : '#ffffff'; } function getFSACount(CFSAUID) { var fsaCount; for (var i = 0; i < fsa_array.length; i++) { if (fsa_array[i][0] === CFSAUID) { fsaCount = ' has ' + fsa_array[i][1]; break; } } if (fsaCount == null) { fsaCount = ' has no '; } return fsaCount; } function getFSAColor(CFSAUID) { var color; for (var i = 0; i < fsa_array.length; i++) { if (fsa_array[i][0] === CFSAUID) { color = getColor(fsa_array[i][1]); //console.log(fsa_array[i][1] + '-' + color) break; } } return color; } function style(feature) { return { weight: 1, opacity: 1, color: 'white', dashArray: '3', fillOpacity: 0.7, fillColor: getFSAColor(feature.properties.CFSAUID) }; } function highlightFeature(e) { var layer = e.target; layer.setStyle({ weight: 2, color: '#333', dashArray: '', fillOpacity: 0.7 }); if (!L.Browser.ie && !L.Browser.opera) { layer.bringToFront(); } info.update(layer.feature.properties); } var geojson; function resetHighlight(e) { geojson.resetStyle(e.target); info.update(); } function zoomToFeature(e) { map.fitBounds(e.target.getBounds()); } function onEachFeature(feature, layer) { layer.on({ mouseover: highlightFeature, mouseout: resetHighlight, click: zoomToFeature }); } geojson = L.geoJson(bcData, { style: style, onEachFeature: onEachFeature }).addTo(map); var legend = L.control({position: 'bottomright'}); legend.onAdd = function (map) { var div = L.DomUtil.create('div', 'info legend'), grades = [0, 1, 5, 10, 15, 20, 25, 30], labels = [], from, to; for (var i = 0; i < grades.length; i++) { from = grades[i]; if (i === 0) { var_from_to = grades[i]; var_color = getColor(from); } else { var_from_to = from + (grades[i + 1] ? '–' + grades[i + 1] : '+') ; var_color = getColor(from + 1); } labels.push( ' ' + var_from_to); } div.innerHTML = labels.join(''); return div; }; legend.addTo(map);
That is pretty much all there is to creating very nice looking interactive free open-source choropleth maps for your Django website application!