Category Archives: Modelling

Pain FREE POINT CLOUDS- Cloud Compare

Big point clouds are here to stay in the Civil world. I feel like Autodesk hasn’t really given us all the tools we need to get the job done. So I want to introduce a great open source program Cloud Compare (download link). Please donate if you start using it heavily.

Cloud compare will really help with the heavy lifting and pre processing of point data before going into the Recap/Civil 3D cycle.

If your feeling lazy watch the video here.

Alternatively read on

Cloud Compare

  1. Drag drop your point data LAS/LAZ/XYZ/CSV straight into cloud compare
  2. Tick off all the things you don’t need. I only leave on Classification, Apply to All
  3. Say Yes to All on the coordinate shift. It will move it back when you export it, so Don’t panic!
  4. Give it a spin and check out those trees (if you have any)
  5. Lets get rid of the trees, select the clouds in the DBTREE (use ctrl)
  6. Go to Edit – > Scalar Field – > Filter by value. We want to filter the ground points which are class 2 (See Classification here). So the range will be from 2 to 2
  7. You will now have a point cloud with no trees in your viewer. Next we want to reduce the amount of points, we call this thinning.
  8. Firstly check how many points you have in your clouds. You can do this be looking at the properties on the left when selecting the cloud
  9. Your final result you want is between 1 -2 millions points if your are working with the survey to create corridors and volume surfaces. (if your are using gradings you might want it to be less than 500k but we can do this in Civil 3d later.
  10. Edit – Subsample
  11. The amount you reduce the point spacing to really depends on what your smallest ground feature you need to be able to extract from the point cloud. So if you know you have a drain that is only 500mm wide that you need to be able to see, don’t go bigger than 500mm. It’s a bit of a trial and error thing though, but the generally principle is we don’t need points every 5 cm.
  12. The last thing you want to do is merge the file together and export
    Edit -> Merge
  13. File -> Save. Cloud Compare will export what ever you have displayed when you click this. Chose LAS/LAZ and change the extension to LAZ when you save. LAZ is way smaller than LAS. Its also a good idea to note the thinning and class you filtered it to.
  14. Now its just a matter of creating a Recap Project and drag dropping this LAZ file in there. Once your done hit save and go over to Civil 3D.
  15. You can read in a RCP file from the Insert Ribbon – > Attach Recap Point Cloud or Type POINTCLOUDATTACH
  16. After you attach, one of the first things you want to do is change your viewframe back to 2D wireframe. It will make the point cloud disappear and replace it with text but you can still use it and you wont get the performance lag you get in 3d wireframe
  17. Now we can select the point cloud and chose “Create Surface from Point Cloud” from the ribbon
  18. Give it a good name and include the capture date of the survey EG-SURVEY_YYYY-MM-DD
  19. Do your self a favour and put a link in the description the the source files you used to create the surface as well.
  20. Don’t click create surface yet. You can now filter the points down more or clip to box only first. But we have already filtered it down pretty good in CC. As mentioned before make sure you are less than 2M
  21. Lastly Make sure you change the filtering to “No filter”. The filter is only used for Unclassified point clouds or photogrammetry points that have trees/buildings and cars in them.
  22. You should now have a surface in Civil 3D that is performance friendly
  23. Do yourself a favour and data shortcut it out. Then data shortcut it in to a separate design file before working.

Loading

QGIS – Creating Longsection Profiles along lines at a specified interval

There are many analysis applications that require long section data or profiles to perform 2d sectional computations. Getting this data seems to be tricky for a lot of these users, so in this tutorial I want to share a method that’s free and easy using QGIS (2.18.4 when I wrote this).

Watch the video here if you’re feeling lazy.

Prerequisites

  • Vector Data for lines (SHP file, Excel Start/End Coords)
  • Raster Data (Geotiff(s)/Esri Grids etc.) contain desired elevation data. You can use multiple rasters if desired for geology etc,
  • QGIS Version >2.18
  • “Locate Points Along Line” – Plugin Installed (demonstrated in video)

Process

In the image below we have a series of lines that we wish to extract elevation data along.

Typically we will start with data that looks like below, its really important that we have at least

  • Easting and Northings of Start/Ends or vertices.
  • A group field to determine which points belong to which lines
  • And an order field to determine which direction the lines is drawn in.

Save this file as a csv somewhere

Now click the big comma on the left to load a csv in QGIS

Ensure “comma” delimiter is checked, “First Record has field names” is checked. (typically it does) and that the X field and Y field are choosing the correct column for geometry, this typically happens automatically.

Finally check the output at the bottom to determine its write.

Very important to note here that you will need to then specify the coordinate system(CRS) this data is in. Right click the layer and select “Set Layer CRS”

Type in the filter to find the correct CRS

All the points have come in but we now need to connect the dots

Open the Toolbox (Ctrl+Alt+T)

Search for “Points To Path” in the toolbox and double click to load

Specify the layer that contains the points, ensure the group (line ID) and order(numbers to determine line direction) fields are selected correctly. Then chose a place to save the shp file.

We now have lines but we need a raster to get elevations from

Either “Drag-Drop” in the Geotiff/Esri Grid etc. you have or select the add raster button on the left

Double check the raster covers the extent of the lines! You may need to assign the CRS for this later as well.

We now have everything but the points at the desired interval to get elevations from. We now need the plugin “Locate Points Along Lines”. Install from “Manage and Install Plugins…” if you don’t have it!

Choose a name/interval spacing and ensure “Keep Attributes” and “Add Endpoints” are ticked on. Then hit run.

Check that you have the desired results and the end points

Finally we need to add the elevations to the attribute table of these points. Right Click on the layer and select “Attribute Table” and you will see current there is no elevation data column.

In the toolbox again search for “Add Raster Values to Points”

Ensure you select the 0.5m interval points layer you created. Chose the rasters to process and also chose a place to save the resulting layer.

Remove the old layer that doesn’t have elevations

Right click on the finished layer and select “Open attribute table”

You should be able to see the elevation column at the end.

Select the top left corner of the attribute table and the press “ctrl+c” to copy all the data out

Then just past in excel and clean as required

Loading

Creating Custom Parameteric Pipe Structures in Civil 3D Part Builder

Its not everyone’s favourite part of Civil 3D. But I get a lot of questions about it.

Civil 3D Part Builder has the ability to create custom parametric parts of all shapes and sizes. The video below will explain everything in detail, the rest of the blog further below will just run over the stumbling blocks you need to know.

Adding Custom Contexts

To add custom contexts you will need to go to this path and edit the “AeccPartParamCfg.xml”. You will need to restart civil 3D to see the changes in part builder

C:\ProgramData\Autodesk\C3D 2016\enu\Pipes Catalog\Aecc Shared Content\AeccPartParamCfg.xml

Add your custom parameter in the <AeccParamDeclaration> section, just copy another one and change it to suit.

Then add your same context into the <AeccParamUsage> section further below. Make sure you enter it under the correct structure type.

Linking the Custom Context to the Dimensions

To create the link to the custom context parameters you will need to ensure your part is mostly finished first. Ensure you have a dimension in Civil 3D that represents the custom parameter, you will override it in the part’s XML

Now open the parts XML file it can be found in the catalog location here.

IMPORTANT NOTE: opening the part in part builder will reset these changes shown here and you will need to go and do them again

C:\ProgramData\Autodesk\C3D 2016\enu\Pipes Catalog\ANZ Metric Structures\Junction Structures with Frames\AeccStructRectSlabTopRectFrameANZ_Metric_offsets.xml

Inside the XML you will need to find the dimension that represents the basepoint offset width.

IMPORTANT NOTE: opening the part in part builder will reset these changes shown here and you will need to go and do them again

So this is that the line should look like

Adding Custom Sizes

You can add custom sizes in this same XML.

Find the parameter you want to change and copy the lines down. Ensure you update the index number as well.

Updating the Catalog and Validating

When your done making changes you will need to update the catalog in civil 3D. Command PARTCATALOGREGEN, select structures or pipes, whatever you need.

I also recommend validating the parts in the catalog screen.

The results can be found here

C:\ProgramData\Autodesk\C3D 2016\enu\Pipes Catalog\ANZ Metric Structures\Validate

Find your part name and open the txt file.

Everything Else!

I suggest watching the video above.

Enjoy!

Loading

Importing/Preparing Australian IFD data as IDF for SSA analysis

Get a few people asking me this one so here you go

Download  Australia Format IDF for SSA using this link <<

Feeling Lazy, watch the video.

Otherwise, firstly you will need to set your project to a hydrology method that utilizes IFD’s. (‘Merica calls them IDF). Go to Input -> Project Options

Change the Hydrology Method to Rational

Now you need some curve data. Google Search, “Australian IFD data” or go to this link for the bom website. You will need to type in the coordinates of your site. Then you can copy or download the data as a CSV.

“IDF” curves in SSA are formatted with the AEP or “Year Return Values” as the rows and min. duration as the columns so you will need to transpose your data in excel. Quite easy, select it all and then rightclick -> paste special -> transpose

You will then need to import this into SSA. I have saved you some time here, you can download the default format here. (this arranges the AEP/duration columns correctly.

Go to Input -> IDF curves

You will want to load my “Australian IFD Format.idfdb” file (see download link here) so that is structures the AEP/duration correctly.

Now just copy/paste the duration data from excel into the little panel down the bottom.

You should now be able to see your curves you can use for analysis

Loading

Dynamic Drawing Production Cheats for Civil 3D Objects – Slope Shading and More

Why we all eagerly await the day that all Civil 3D objects, like corridors, become data shortcuts or Civil 3D gets replaced with Infraworks (Not sure which is going to happen first). Until that day though we still need to get those Civil 3D models into our drawings in the most dynamic way possible. So here are a few sneaky tricks to do so, keeping everything as dynamic as possible and not exploding a thing.

  • Surface Slope Shading
  •  Surface Boundary Technique for Linework and hatching
  • DXREF’s data shortcuts in XREFS

Feeling Lazy watch the video. (Coming Soon)

While I love all the slope patterns that get generated from grading’s and corridors. I’m sick of exploding AECOBJECTEXPLODE all my linework from my model and copying it across to my xrefs etc. Every time I update, drives me crazy!

So what If I told you that the image below has not a single bit of exploding in it. Everything is produced via data shortcuts, even the linework, all dynamic. Interested? read on ->

Slope Shading
It’s so simple you will kick yourself for not working it out earlier. To produce the grey slope shading on the surfaces is quite simple. Using the surface slope analysis. Right Click – Surface Properties – >Analysis Tab – > Select Slopes from dropdown.

My embankments are 1:2 cut and 1:2 fill -> so >40% and my road cross fall is 3 %. So im going to drop down two ranges

  • 0.0001% – 10 % and (< -you can’t type 0 exactly)
  • 10% – 999999999999999% (type lots of nines in case you make even steeper areas than you have, then it always updates)

Change the colours to 254 (light grey) and 253 (Slightly darker grey) respectively.

image010image001

Then you should have what I have above, dynamic updatable surface shading. If you can’t see anything then you will need to choose a surface style that has the slopes turned on under the “display tab”. See below. Make sure you set a layer for the slope objects if you want to utilize Autocad transparency.

Linework and Hatching – The Surface Boundary Technique
I unfortunately have to assume that if your reading this you know how corridor point, link and shape codes work. What we can do with these wonderful codes is create surfaces where the surface boundary represents the linework we are trying to display on plan. The Red, White and Orange dashed lines taht represent that linework.

So I create 3 additional surfaces in my corridor properties.

  • DS-ROAD_curb,
  • DS-ROAD_footpath and
  • DS-ROAD_pavement

You will then need to work out the boundaries for them. You can do an interactive boundary if you feel like wasting hours of time. Or just use the Angle/Length Trick.

Tod do so, go to the surface properties for the footpath for example. Definition Tab – > Expand the Build Section and make the maximum angle like 92º and the maximum length about 5m.

To explain what this is doing. It will analyse the footpath surface for triangles that have an angle greater than 90°, quite often these are the weird “spiderwebs” we get on the outside of the surface. Then it will analyse the triangles that have a length on any side greater than the number specified and remove them. It’s important to note that the “Maximum angle” only works on outside triangles, not internal triangles. The “maximum length” will remove triangles anywhere. See the diagram below.

Before Applying Angle/Length Parameters

After Applying Angle/Length Parameters

The finished result is just footpath. You can use this trick on all your surfaces though

Now do the rest of the surfaces as well in a similar fashion. You will need to create a style that has only border displayed to see the finished result. You may also need to play with your corridor frequencies to get the desired result.

Creating a surface style that only uses the border object

The finished Result is dynamic linework though. Using a surface border.

Surface that represent linework only using surface border styles

As for hatching well its as easy as using that hatch command

Putting it all together

The final step is to collate all this together for use in a drawing. Here is my recommended workflow.

  1. So i have data shortcutted my top surface and set the shading.
  2. Data shortcutted my linework surfaces in and created different “border only” surface styles that I needed.
  3. Added a hatch onto the pavement surface.
  4. I also brought in my alignments and station/chainage labels in as well.

Now I simple select it all and WBLOCK it out to an xref location.

The best part about the WBLOCK is that you only have the layers you need in the new drawing. Meaning improved performance when using it as an XREF as well as not doing your head in. WBLOCK does not destroy the data shortcuts either!

Let me know what you think below.

Loading

Diagnosing the Surface Pasting Problem – “Error Performing Edit”

We all get these from time to time. Unfortunately Civil 3D is rather vague about what is causing this error.

  or  2016 error adding boundary

^^^^ I HATE YOU ^^^^^

I can tell you empirically that the primary cause almost 99% of the time is a boundary that has been added to a surface somewhere along the chain that needs to be cleaned up.

A few people will suggest using things like MAPCLEAN and WEEDFEATURES to fix over complicated boundaries lines and breaklines and these methods do help. But they don’t always work.

A great method for fix this is to use the OVERKILL command. It will remove overlapping segments that are duplicates within a polygon. Commonly this comes from output from other software!

overkill commandoverkill result

If you still have no luck try a method that I have used for years. Simply “STEPPEDOFFSET it out and then STEPPEDOFFSET it back in” maybe 0.5m or so.. (note: occasionally you may need to offset it back in 1 mm different, so back in 0.499m)

I have over exaggerated this to explain the issue, but here is what an offset does to a messy line with overlaps and duplicate vertices

Just whatever you do, don’t go extracting the triangles and re-adding them to a surface!!!!

Loading

Stage Storage Analysis of Stockpiles/Dam Walls (between two irregular surfaces)

UPDATE: You can do this even easier with the “Stage Storage Basin Tool” available in my free tools package here
https://ceethreedee.com/ceethreedee-tools

It is quite easy to do perform a stage storage analysis of a simple basin or dam water. This is because all the volumes are compared to a flat elevation, one flat surface (water level), one irregular surface(ground surface).

But what if you want to know the stage storage elevation of a dam wall or a stockpile, i.e two irregular surfaces).

Well you can essentially use the same technique as described here, but you will need to compare the two stage storage analysis’s to get the final answer.

Keep reading to find out how or if your feeling lazy watch the video here.

Basin Stage Storage (Flat to Irregular)

image004

Stockpile Stage Storage (Irregular to Irregular)

image006

By turning our two irregular surfaces into two separate stage storage calculations to a common flat surface, we can then subtract one from the other to form a stage storage of the stockpile.
image009

Firstly we need to create the common flat surface, the best technique is to find the Max RL of the design surface, by looking at the Surface Properties.

image013

On this stockpile the max RL is 1560 . So, I’m going to create a flat plane at 1560 m and add it to a new surface called “CALC LEVEL”

image019

Now we need to limit our calculation to the design extent
image021

So you will need to create a surface that clips the existing ground to the design surface boundary.
image014

Now you have all the surfaces you need, you can create the two volume surfaces to do the stage storage,

  • one that compares to the design, – The Subtract Surface
  • and another that compares to the existing ground – The Addition Surface

image023

Extract the stage storages for both of them using the technique described in my post here

image016

You will now need to do a simple subtraction between the two stage storages to form a combined stage storage like below.
image017

The finished result it is quite clear. Looking at the area it starts narrow at the bottom, increases to full width where the toe meets the ground, then reduces in size as it goes back down to the crest of the stockpile. The change in volume also reflects with the increase in area.

image020

Do a final check with a volume surface at the end to see if your correct.

Loading

Managing Surface Volumes in Civil 3D – Excel and the Volumes Dashboard

We are often exporting volumes from our models and for a lot of designers and drafters it is an adhoc affair. Engineer asks for volumes, so you make a few volume surfaces quickly, export volumes into excel, format a bit and then send. Some of the slightly more organized types might even save the excel spreadsheet into their modelling folder in case they need to use it again! But more often than not you will be asked to do this several times over the life of the project. So why not come up with a decent system for handling this process?

 

I would like to present a more efficient method, that not only is easy to update and manage but will improve the performance of your modelling files as well.

The workflow is quite simple, it may take slightly longer to setup. But I promise, you will thank yourself later. They key element been the use of the volumes dashboard.

A few simple rules to follow to ensure you maintain a nice link.

1. Never create all your volumes in the modelling DWG. Create them all in a separate DWG using data shortcuted surfaces (The odd one is fine).

2. Don’t manipulate the raw data you copy from the volumes dashboard (other than a sort). Link to it from another table in excel.

3. Don’t “Cut data over”/ Copy Data over the top in excel. Otherwise your links break!

Volume surfaces quickly increase the size of modelling dwg’s and will decrease overall performance when working in them. Unless you don’t work in your modelling DWG’s very often, which I highly doubt, keep your volumes surface in a separate dwg and data shortcut all the surfaces you need in. This also makes it easier to extract all the volumes you need in one hit rather then prowling around your dwg’s looking for quantities.

Once you have created your volumes DWG and you have data shortcutted the surfaces you need in. Start creating Volumes Surface Pairs. I strongly suggest maintaining a simple but consistent naming system. My system is as follows

<identifiers>-<object>_<material_name>

So for example:

OP2-NTH-ROAD_gravel

OP2-NTH-ROAD_base

OP2-NTH-ROAD_ subbase etc..

This makes your life a lot easier when creating links in excel. It also helps prevent the desire to shoot yourself due to confusion later on :p.

Now you have pairs, it’s time to setup an excel file. My pairs looks like so:

Right Click on any surface in the dashboard to “Copy to Clipboard”

Open Excel, an begin by creating two tabs “Volumes”,”Data”. Data is where we will copy the raw data from your volumes dashboard (copy your raw output now). Volumes is where we will create the links to the volumes dashboard data so it is easier to update later.

image002

Your data is now on the data tab. Do a sort on it (Volumes Dashboard doesn’t have a sort function!)

Your data should sort by surface name! This will save headaches later when you want to update the table or add new surfaces.

Now switch over to the Volumes tab and start creating a table like the one below. With various identifiers you need for each material volume you want to calculate. In the last column (2nd last in my case) you want to link the QTY back to the “data” tab that contains your raw pasted volumes dashboard data.

Now this table you create will eventually contain all the volumes you want, but not exactly in the best format for visualizing. This last step can be quite confusing for a lot of people. So hopefully the video link at the top helps.

Because the table is arranged in a “basic record format” like Microsoft Access or any SQL etc. we can use this table to create “Pivot Tables” and “Pivot Charts”.

Firstly covert your table into a “Named Range” by using the format as table function

Now select anywhere in your nicely formatted table and go to Insert -> Pivottable.

You will be confronted with a blank screen and most likely see all your column headers on the right.

By drag-droping your tables headers into the fields as required you can make your table display exactly as you need it. (I highly suggest watching the video to see this in action)

But for example say I wanted a table to display the materials as the column headers, the time as rows and calculate the quantities with respect to these columns. I could do the following

Then you will get this

But you can adjust these to suit your needs. Better, you can adjust to suit other people’s needs. You can create as many pivot tables as you want to display different views of your data and they are all stay linked back to the original source..

Now when it comes time to update your volumes from Civil 3D all you have to do is copy your volumes dashboard output… sort and paste over the top in your data tab!

The end result wonderfully formatted, dynamic tables. That can update almost at the click of a button…

Component Time-Height Embankment Storage Decant Rock Floor Cut Floor Fill Wall Excavate Windrows Minewaste Dry Tailings
3 – NORTH Month 00 – 487.82 m – DOWN 16,600 500 5,900 5,200 6,100 500
Month 06 – 487.82 m – DOWN 39,900 500
Month 06 – 489.92 m – DOWN 27,700 500
Month 12 – 493.02 m – DOWN 116,500 60,800 1,000 600
Month 21 – 493.02 m – DOWN 101,500
3 – NORTH Total 160,800 102,200 2,000 5,900 5,200 6,100 1,600
4 – SOUTHEAST Month 06 – 489 m – DOWN 125,100 600
Month 18 – 489 m – DOWN 325,100
4 – SOUTHEAST Total 125,100 325,100 600
5 – MINEWASTE Month 18 – 497.7 m – DOWN 545,600
Month 24 – 497.7 m – STACK 405,900
Month 30 – 497.7 m – STACK 600,800
Month 36 – 497.7 m – STACK 390,700
5 – MINEWASTE Total 1,943,000
6 – DRY STACK Month 12 – 485.2 m – STACK 190,600
Month 18 – 485.2 m – STACK 45,100
Month 24 – 487 m – STACK 47,200
Month 30 – 489 m – STACK 46,000
Month 36 – 491.8 m – STACK 52,300
6 – DRY STACK Total 190,600 190,600
Grand Total 285,900 527,300 2,000 196,500 5,200 6,100 2,200 1,943,000 190,600

Loading